uSens Reveals How Its VR Hand Tracker Uses A.I. to Ditch the Touch Screen

Ready to reach out and grab virtual objects with your hand? uSens announced the beta release of its uSensAR Hand Tracking software development kit at the Augmented World Expo in California on Thursday, enabling iOS and Android app developers to build hand tracking into their creations. While augmented and virtual reality allows users to explore visual worlds with their eyes, this breakthrough could take that interaction to a whole new level.

“Imagine developers building AR applications where they can use the hand to interact with virtual objects,” Yaming Wang, vice president of product and operations, tells Inverse. An AR app that lets users preview furniture in their home, for example, could enable users to move tables and chairs around by using their hand to grab them — bypassing the touchscreen entirely. “We see a lot of potential applications.”

The software is set to reach millions of users soon. uSens is working with a number of major firms including Meitu, a selfie-taking app with a staggering 455 million monthly active users, to add new augmented reality features into the app. Details are yet to be announced, but Wang tells Inverse that the front and back cameras can be used simultaneously to enable both selfie-taking and more social interaction at the same time. It’s a cool application for the tech — but it’s not the only idea that the team came up with.

“We were planning to build a demo,” Wang says, describing a whack-a-mole-style game. “Today, if you play on the phone, you use a touchscreen. It’s not very intuitive. Imagine using your own hand as a hammer to knock out the moles.”

Smartphones are just one step on this journey, though. The company has developed Fingo, a hand tracking hardware add-on for virtual reality headsets that means users don’t have to fiddle about with stick-shaped controllers. Big names like HTC have expressed interest in getting hands into their future products.

“In the short term I think they’re going to coexist, but definitely customers we’re talking to all have the intention of replacing controllers with hands,” Wang says. “That’s definitely the direction it’s moving to, and a lot of headset companies are moving in that direction.”

This hand tracking isn’t just for the iPhone X and the like. The technology was originally designed for use with Spreadtrum-powered phones, a Shanghai-based chipset maker that targets low-end devices around the $100 range. As long as it has an RGB camera and a supported chipset, the software should run smoothly.