Purdue Researchers Develop Furion® — A New VR Framework Enabling High-Quality Virtual Reality on Mobile Devices

TL; DR: With the relatively recent availability of mass-market virtual reality (VR) headset devices, there’s no denying the technology is here to stay. However, according to Y. Charlie Hu, Professor of Electrical and Computer Engineering at Purdue, VR must be readily available on mobile devices for it to become a true part of our everyday lives. And, now, researchers at Purdue are realizing this dream with Furion. The novel software platform allows untethered access to high-quality VR from existing smartphone technology. As Charlie said, Furion represents the first step in exploring the next frontier in virtual reality — mobilizing the technology with a quality of experience (QoE) never before seen.

Since Pokémon Go first appeared in the marketplace, players have flocked to the game in droves to embark on GPS-guided journeys to capture virtual monsters in real-life places — from Charmanders on countertops to Spearows on sidewalks. However, gaming fanatics have learned the hard way that, when it comes to augmented reality (AR) encounters with Pokémon creatures, it’s impossible to “catch ‘em all” with a dead smartphone battery.

While the hyper-popular app has been revolutionary in bringing the AR gaming experience mainstream, it’s extremely taxing on today’s mobile devices.

“Pokémon Go is one of the most power-hungry apps today,” said Y. Charlie Hu, Professor of Electrical and Computer Engineering at Purdue University, noting that the game can render a fully charged smartphone lifeless in just a few hours.

But, considering that AR games, such as Pokémon Go, merely superimpose digital objects over a user’s view of the real world (as seen through a smartphone camera), they’re nothing compared to the fully immersive virtual reality (VR) experience in terms of battery drain. High-quality VR games are so much more power-intensive that they require a special device tethered to a server or PC, which performs the heavy rendering load by cables that prevent mobility and present tripping hazards, Charlie said.

He and other Purdue researchers are aiming to free users of such constraints through the development of Furion, a revolutionary software that uses cooperative rendering to reduce strain on smartphones, enabling them to support VR. The new technology, a product of the researchers’ quantitative design study on the feasibility of high-quality VR on untethered devices, even satisfies industry standards for quality of experience (QoE). According to Charlie, the software will be key in motivating developers to create more content, serving a wider audience of VR gamers and ultimately jumpstarting industry growth.

Why Better Hardware Performance & Wireless Networks Aren’t Solutions

By studying the current state of high-quality VR apps on modern smartphones, Charlie and his team discovered even today’s high-end mobile devices, such as Google Pixel, are about 10 times less powerful than what’s needed to achieve acceptable VR QoE. But, according to Charlie, waiting for advancements in hardware and wireless networks simply won’t help.

“It’s not about hardware technology limitations,” he said. “We can certainly make the hardware in your smartphone 10 times more powerful to support high-quality VR on your smartphone; technology-wise, we know how to do that.”

While the technology exists, Charlie said it’s not ready to be implemented for a practical reason: lithium ion battery constraints.

“The capability of the smartphone has pretty much tampered off simply because we have not been able to invent a better battery,” he said.

Until we have a better battery, more powerful phones on the market will simply drain batteries at an unsustainable rate, according to Charlie.

He said it’s also unlikely that waiting for the advent of next-generation wireless networks will make a difference in facilitating untethered VR experiences on mobile devices. According to the group’s study, “the multi-Gbps bandwidth promised by the next-generation of wireless networks will not make remote rendering readily feasible because packet processing at a 10 times higher data rate will far exceed the mobile CPU capability,” which is, again, limited by battery constraints.

Spurring Innovation by Motivating Software Developers

At the same time, the VR industry faces a cyclical dilemma. Consumers are hesitant to buy VR devices, and developers aren’t particularly inspired to create games for units that don’t sell.

“So far, there have been fewer than 1 million or so smartphone [VR units] sold, and 1 million is a small number considering that we have 2 billion smartphones,” Charlie said. “Hardware sales have not picked up quickly because of the lack of VR content. You don’t want to buy special VR hardware gear just to play one VR game. You are much more motivated to make the investment if you have unlimited VR apps you can purchase and play.”

With little incentive for content creators to develop VR apps, the industry becomes stagnant. That’s where Charlie’s team comes in.

“Imagine if we can improve the quality of VR on [high-end] smartphones,” he said. “Then, suddenly, we can break this dilemma. The VR content creator is much more motivated to create new VR content because there is a bigger market, and, hopefully, that will foster special VR hardware gear sales.”

Bringing VR to Wider Audiences Through Cooperative Rendering

To that end, Charlie and his team have designed Fusion, a new type of software that employs a cooperative rendering approach to divide foreground interactions from the background environment. The technology makes high-quality VR possible on current mobile devices and wireless connections.

“This was more about engineering than radical and new ideas,” Charlie said. “The main IT solution idea is offloading.”

And, while offloading as a concept has been established for quite some time, the team developed a totally new way of applying it.

“Our key insight is that we can separate the rendering of the foregrounds from backgrounds and then offload the load-intensive rendering of the backgrounds onto the much more powerful PC over wifi,” Charlie said.

The background environment can be offloaded to the PC because it doesn’t fluctuate as often per frame, and the changes that occur can be pre-rendered and pre-fetched since they are relative to the player’s location and, thus, predictable. The mobile device is responsible for rendering foreground interactions only, drastically reducing the load on the device. The two are then combined on the mobile device to generate the final frames.

For developers working on high-quality VR apps, the approach opens a door to a whole new pool of customers.

“Their customers are no longer limited to those who possess the current VR hardware gears,” Charlie said. “They suddenly have access to a much broader audience.”

Providing Superior QoE with 60 FPS and Low Latency

Hu and his team were able to quantify Furion’s success in enabling high-quality virtual reality on today’s smartphones by determining visual quality, frame rates, and latency to determine users’ potential QoE.

“The virtual reality industry is well-defined by two specific measurements: latency and frame rate, which has to be at least 60 frames per second,” Charlie said.

By splitting the foreground interactions from the background environment, the 60 FPS requirement is met with a frame-rendering time under 14ms, which ensures QoE.

That milestone, however, only applies to individuals.

“So far, we have really only solved the problem for the single user, Charlie said.

With that in mind, Charlie and his team hope to overcome bigger hurdles, expanding the split-rendering power of Furion to allow for multiplayer support, which demands even more resources.

“We are currently expanding that solution to support multi-user VR games,” he said. “It’s going to be much more of a challenge, but that’s what we’re working on next.”