The Future of VR Means Tracking Your Eyeballs

Like it or not, virtual reality hit mainstream this year. If you own a PC, you have access to full virtual reality experiences, complete with motion tracking and room-scale movement thanks to new commercial headsets like Vive and Oculus. Not to be outdone, Sony also recently released PlayStation VR, bringing a formerly high-end PC niche to all 40 million-plus PS4s that have already been sold. And while it’s still unknown whether VR has staying power, one company is betting on an approach that seems even closer to sci-fi: a headset that tracks your eye movement.

FOVE (based on the eye’s fovea, a small depression in the retina) went all in with the concept last year — to the tune of about $480,000 raised on Kickstarter, plus some interest from investors — to build a VR headset specifically designed to read eye data. Due out next year, the so-called FOVE 0 deviates from traditional VR by using a specially adjusted infrared light, in that it’s invisible to the user. This illuminates the pupils from up close, then uses sensors inside the goggles to read eye movement as well as the direction and focal point of a user’s vision.

The headset also uses what’s called foveated rendering, which basically works like a virtual depth-of-field effect, giving the object of a user’s focus staggering detail while offsetting the focus elsewhere. Coupled with the traditional head-tracking tech found in most every VR headset, what FOVE is gunning for is an entirely new level of realism in applications running on the FOVE 0.

“Eye tracking is an additive layer that makes interacting with a VR environment incredibly fast and natural,” says Yuka Kojima, FOVE’s co-founder. “Head tracking and input devices are still needed, though — the headset only knows what you are looking at and can’t determine your intention. Having a physical button to push allows you to simply glance at an icon or object and use the controller to take action immediately.”

Kojima, who founded FOVE with business partner and CTO Lochlainn Wilson in 2014, first envisioned the idea while working at Sony.

“The most significant influence [I had] was the desire to have genuine and human-like experiences with the characters that populate our game worlds,” says Kojima, who is largely responsible for FOVE 0’s look and feel as well as the company’s business strategy.

“I wanted to create a world where I can spend hours with these characters and really get to know them,” she says. “For example, I’d like to build a world with a virtual character and just as we’re about to dive into the final boss battle, we can lock eyes, creating an emotional experience.

Expression, then, is what drives FOVE’s company vision. It’s not enough to have players existing in another world — Kojima wants them to feel.

“We want to bring human emotion to VR,” she says. “Currently, players can only communicate through a text window in games, but I want them to be able to do so with their eyes, and someday even laugh together through [common] expression.”

Though you might not realize it, a headset design like the FOVE 0 gathers a lot of information about a users eyes; it’s much more than simply where you’re looking or focusing.

“It’s not as much about eye movement in VR as it is about eye data,” Kojima says. “Your eyes are constantly darting around, in what are known as saccades. Only when you are focused on something specific do your eyes lock onto it. This is what makes typing with your eyes so difficult — you don’t smoothly flow across a keyboard.”

If building a regular headset for VR is hard — which already includes engineering, head and motion tracking, maintaining 1:1 latency and hitting the right resolution and framerates to keep users from getting sick — adding pupil tracking to the mix presents its own challenges. For one, working with eyes means running into biological differences that wouldn’t necessarily present a problem with standard VR equipment.

“One of the biggest challenges [we discovered] is the sheer variety of eye and pupil shapes,” Kojima says. “Gender, age, and ethnicity all play a significant role in the width and depth of the eyes, while things like eyelids, eyelashes, and even mascara or eyeliner need to be accounted for.”

Another big issue is what’s called “headset drift,” which is how a headset will slightly change position as you adjust placement, look around, or otherwise react. Essentially, the FOVE 0’s tracking sensors must take movement into account consistently to prevent the headset from having to recalibrate relative to the position of your eyes.

“Every time the headset moves we have to automatically adjust that map to make sure we continue to track your eyes reliably,” Kojima says. “It’s one of the toughest challenges in eye tracking VR.”

All of this has broader implications for gameplay as well.

“Where you eyes are looking is what’s really interesting,” Kojima says. “What if an NPC knows that you’re constantly checking his gun, or looking at a potion on a shelf, or what you’re constantly not looking at? Imagine a [user interface] that tracks that, noticing when users ignore far off [in-game] warning signs, or an algorithm that adjusts them so players start noticing them.”

That certainly sounds promising, but much like a developer adapting a non-VR game to virtual reality, it will fall to game creators to make use of FOVE’s groundbreaking technology. FOVE has thus far showed off the tech in Project Falcon, an on-rails shooter that was shown off at this year’s Tokyo Game Show, though the website notes over 250 SteamVR games work natively with the headset (also it’s still early days for developers working on integrating the company’s tech moving forward).

While Kojima appears most interested in games, FOVE has a host of other potential applications; she mentions eye-based authentication for online shopping, social situations involving virtual eye contact, and VR productivity uses, and the scientific possibilities seem numerous. Above all, what FOVE 0 really represents is, hopefully, a step forward for interaction.

“The future of VR is through social communication and bringing characters to life,” Kojima says. “We want to take communication technology even further and we believe that the next step is expression — and we’re planning to combine our eye tracking with facial tracking to fully immerse people into VR.”

Holodeck, here we come.