Navigating virtual reality: The eyes have it

NEW YORK — First came the computer mouse. Then, the touchscreen. Now the tech industry is looking for a new human-machine interface — this time, one that will make virtual-reality (VR) headsets as mainstream as personal computers and smartphones. The man who invented the LeapPad tablet for kids is betting the killer app is right in front of your face: The eyes.

Mr Jim Marggraff’s start-up, Eyefluence, has developed technology that knows where people are looking and lets them manipulate objects the way we do now by clicking a mouse or tapping an icon. Besides fostering a more natural and immersive experience, the system is designed to help alleviate the nausea experienced by some VR users and enhance security with iris scans.

Mr Marggraff says most of the big headset makers have expressed interest in licensing the technology from his company. Motorola Solutions, a leading investor, is testing the technology for emergency responders and sees possibilities in mining and medicine. “You’re able to basically interact in the virtual world simply by looking at what you want to interact with,” says X Prize Foundation founder Peter Diamandis, who advises Eyefluence and has seen the technology.

Most VR headsets use some sort of handheld device (like a game controller) or head movements to navigate. These techniques are far from ideal because they require a lot of moving around that gets tiring, and industry heavyweights — including Facebook chief Mark Zuckerberg — agree that some form of eye-based technology is key to improving and popularising VR technology. Gaming enthusiasts also say eye-tracking would significantly improve the playing experience by making it easier to follow objects and interact with characters.

Already, a host of companies with names like SensoMotoric Instruments, Tobii and The Eye Tribe are working on eye-tracking devices that could plug into a range of headsets.

Another company called Fove says it will soon have the first headset featuring a version of the technology. But most of these are limited to scanning the iris for security purposes and recognising where the user is looking — pointing but not clicking.

Eyefluence takes it one step further, not only using the eyes as a cursor, but letting them select, zoom, pan — things now accomplished by clicking and double-clicking a mouse or tapping and pinching a touchscreen.

The eyes are the fastest moving organ, capable of shifting 900° per second, which means Eyefluence’s interface software makes it much faster to tell the computer what to do.

The technology makes a two- or even three-step thought process into one: Say, from look at object, move hand to it and tap or click to simply look. “It almost feels magical, like the system knows what you want before you tell it,” Mr Diamandis says. “It’s almost like it’s reading your mind.”

When Mr Marggraff announced plans to turn the human eye into a computer mouse, sceptics said it could not be done. “People told me, ‘Don’t bother: You consume information with your eyes, but if you try to simultaneously use your eyes to control things, there will be a collision between controlling and directing and consuming,” he recalls.

But Mr Marggraff has a deserved reputation for knowing a good idea when he sees one. He came up with the LeapPad even before the iPod existed. Released in 1999, the device helped kids learn how to read, was for a time the most popular toy in the United States and generated more than US$1 billion (S$ 1.36 billion) in revenue within five years.

Mr Marggraff, 58, also created the Livescribe, a smart pen that records audio and syncs it to written notes. Eyefluence co-founder Dave Stiehr, 57, helped build several medical-device companies, including one that sold automatic external defibrillators — gadgets that diagnose and correct uneven heartbeats and are now standard equipment in many movie theatres, aeroplanes and offices.

In 2012, Mr Marggraff bought all the assets of Eye-Com, a research company led by neurologist William Torch and funded by the National Institutes of Health, the Department of Transportation and the Department of Defence. Mr Torch researched everything from fatigue to blinking and had accumulated more than a decade’s worth of data on eyes as well as built technology to track and analyse the organ. Mr Marggraff and his team used that foundation to create an interactive system that uses looks to direct a computer.

It is not as easy as it sounds. Eyefluence’s team had to design a way that feels natural for the eyes to communicate a person’s intentions to the machine.

Looking at an object for a set amount of time to “click” it, for instance, would not work: It is too taxing on the eyes, and it would take too much active thinking to remember to stare for a certain amount of time or not stare for too long for fear of accidentally selecting it. A successful eye-interaction method would have to be just as intuitive as moving a mouse or tapping an app.

In the end, Mr Marggraff solved the problem by embracing aspects of how the human eye works that he had originally seen as a limitation. Even though it feels like our eyes are providing a constant stream of visual information, there are actually interruptions to the feed when we move our eyes to look at or examine something else. In those moments, we are essentially blind and the brain fills in the missing images.

Using these kinds of movements as a foundation, Eyefluence built an eye-machine interface that acts on intentional looks and ignores incidental ones. The company declined to explain further how the software works, citing concerns about maintaining a competitive edge.

Eyefluence is hammering out licensing deals with various headset makers to include its hardware and software in their devices, though the company declined to discuss specifics, citing non-disclosure agreements.

Mr Marggraff did say that the company is working with the “major players” and has been “overloaded with the strong amount of business partnerships and interest”. Bloomberg