Here's Magic Leap's Covert Vision for Its Augmented Reality Headsets

You may have heard of Magic Leap, future purveyors of “cinematic reality” (augmented reality) headsets. This is a company without a single commercial product, and yet it’s worth $4.5 billion. Many are invigorated by the company’s potential, but a tour of its most recent, 160,000-word, 350-page patent application — especially with current events in mind — could suggest other, less exciting applications for the emerging technology.

No one really knows what to expect from the company: by design, it’s mysterious. (Out of silence comes allure.) We’ve seen two videos, we’ve investigated its job postings, and we know that wealthy organizations love throwing money its way. But we do have access to Magic Leap’s patent applications, and it published its most recent one in late January.

People are excited about VR and AR. With reason, too: the demos, previews, and new releases we get to watch are striking. The games we imagine playing in the future are enthralling. The mere idea of a virtual or augmented world, either distinct from or mixed with the actual (boring, depressing, what-have-you) world, is enough to invigorate all of our wayward souls.

The augmented reality future — seen through a Magic Leap headset — may, for all its irresistible pleasantries, seem a little disturbing.

Let’s take a tour of a few of the potentialities for its technology. (Keep in mind that the patent application itself is phrased in the hypothetical: Magic Leap is still attempting to maintain some air of secrecy about its projects.)

Judging Your Productivity, Lifestyle Habits, and Behaviors

These headsets “are designed to be worn persistently.” Granted, they’ll probably be so much fun that you’ll never even consider taking them off. (Just acknowledge that, fifteen years ago, you never thought you’d be addicted to a phone, then reflect upon how much more thrilling and life-enhancing a highly-functional, beautiful AR headset would be.) But, if you do keep them on throughout your days, they’ll be gathering more data than your smartphone could ever dream of attaining. Behold:

“The AR system may be used to gauge productivity of employees at a company. The company may collect data on an employee’s work habits and productivity and may be able to accordingly provide incentives or compensation to the employee based on the observed productivity.”

And more:

“Because the AR system is constantly capturing images and videos, the AR system may detect unproductive behavior (e.g., unrelated internet browsing, low productivity, etc.), and may appropriately render virtual content to help motivate the user.”

These headsets may also serve as covert healthcare company spies:

“Given the AR system’s ability to constantly monitor a user’s behavior, companies may be able to gauge a user’s health based on his behaviors and accordingly price insurance premiums for the individual. This may serve as an incentive for healthy behavior to drive premiums down for insurance because the company may see that the user is healthy and is low-risk for insurance purposes. On the other hand, the company may assess unhealthy behavior and accordingly price the user’s premiums at a higher rate based on this collected data.”

If you thought you were already being judged enough, think again: AR headsets may be equipped with the capacity to judge and react to your behaviors. And not just your behaviors in the virtual world; your behaviors in the actual world, too:

“The AR system may render a plant that blooms when the user exhibits ‘good’ behavior and wither away when the user does not. In a specific example, the plant may bloom when the user is being a good boyfriend, for example (e.g., buys flowers for girlfriend, etc.) and may wither away when the user has failed to call his girlfriend all day.”

It’s an understatement to say that, despite the best efforts of philosophers, disagreement over what constitutes “good” and “bad” behavior still runs rampant. If Magic Leap headsets so judge our behavior, the judgments will, presumably, be coded. That means that programmers set the parameters of good and bad behavior, coders who, in all likelihood, have probably never read a lick of philosophy.

Now let’s imagine — just as a sort of thought experiment — that Magic Leap decided to involve artificial intelligence in this game. The coders set the initial definitions of good and bad behavior and then repeatedly point out instances of behavior that qualify as either “good” or “bad.” The A.I. begins to “learn” how to “distinguish” between “good” and “bad” behavior, and then all of a sudden we have A.I. overlords. (Self-driving cars “learn” much in this way: you show them a plethora of situations and the A.I. learns and perfects how it will react.)

Your Eyes Told the Whole Story

Magic Leap headsets, the patent application states, may be able to authenticate you based on your retinal signature. This will be a useful feature: “Since the display device projects directly onto the user’s retina, only the intended recipient (identified by retinal signature) may be able to view the protected content.” It will also be a very Minority Report-esque feature, a feature that an authoritarian-leaning government would drool over.

Other possible — and, it would seem, likely (given current trends toward monetization) — applications are advertising, purchasing, and something Magic Leap calls microtransactions. Here are the patent’s authors:

“The subject AR systems are ideally suited for assisting users with various types of transactions, financial and otherwise, because the AR systems are well suited to identify, localize, authenticate, and even determine gaze of the user.”

And those microtransactions, monetizing each and every thing you do with your eyes.

“… Micro-transactions may be easily performed through such a system. For example different products or services may be priced at a fraction of a penny (e.g., a news article may cost 1/3 of a cent; a book may be charged at a penny a page; music at 10 cents a listen, etc.). Within the current currency paradigm, it is hardly practical to utilize micro-transactions, because it may be more difficult to keep track of such activity amongst users. However, with the AR system, the AR system may easily determine the user activity and track it.”

Targeted advertising that makes the Google ads that follow you around the internet seem quaint:

“… The AR system may be used in advertising applications as well. For example, the user may see a particular advertisement on TV, or maybe see a pair of shoes he/she may like on a peer. Based on the user input (eye gaze, touching, or any other input), the user may be directed to the company’s webpage, or to another seller who may be selling the item. For example, virtual icon may automatically populate within the field-of-view of the user, providing various purchase-related options to the user.”

Let’s talk about eye tracking. Sure, you’re blasé when it comes to government agencies reading your emails and listening to your phone calls. After all, what would you be hiding? But what about a technology that — in addition to everything else it has access to — compiles and begins to understand what you tend to look at?

“… The AR system can track eye pose (e.g., orientation, direction) and eye movement of a user, and construct a “heat map”. A heat map may be a map of the world that tracks and records a time, frequency and number of eye pose instances directed at one or more virtual or real objects. For example, a heat map may provide information regarding what virtual and/or real objects produced the most number/time/frequency of eye gazes or stares. This may further allow the system to understand a user’s interest in a particular virtual or real object.”

Mind Control & Virtual Experiences

While there are some exciting applications that would fall under the blanket of “mind control” — a Magic Leap headset, for instance, “may be used to detect seizures, and may also be used to prevent or treat them” — there is one particular sentence that stands out in the behemoth patent application:

“There may be other applications of using light signals that are projected into the retina. This light therapy may further be used in psychological applications, and subtly controlling brain signals to change the user’s thoughts or impulses.”

And then there are the virtual experiences. Depending on what sort of person you are, you may be either repulsed or enthralled by the following applications.

The patent application lays the groundwork for the demise of the physical concert: “… The AR system may render the concert, performance, or other event as a virtual space, mapped onto a user’s physical space.”

Here’s another such application:

In some embodiments, the AR system may be used to shape a pre-existing generalized model of a human … by morphing a set of control points extracted from a data cloud of another person. Thus, the AR system may use a 3D model … of a person’s body, but sculpt another person’s face into the 3D model. … The simple scan may provide enough information about the user’s face to make the generalized model resemble a particular person in fine detail. In other words, the AR system can benefit from the highly precise 3D model and supplement it with necessary detail captured from the simple scan to produce an accurate 3D version of the person.

And, finally, the ability to relive your drunken debauchery the next morning:

“The party may be captured as a full light field experience of the whole party. This allows going back and reliving the party, not as a video, but as full point of view experience. In other words, a user would be able to wander around the room, seeing the people walk by the user, and viewing the party after the fact from essentially any vantage point.”

For the patent’s full text, look here. Be warned: it’s long.