At Sundance, virtual reality defied categorization

Noam CHOMSKY_AI had been acting up. After 10 days of talking to eager strangers, the consensus was that he was overtired. Possibly overheated. But a few hours of debugging later, I'm given a thumbs-up and handed the virtual reality headset. I find myself standing with fog at my feet in a craggy landscape. A metallic blob hovers ahead. “Hi Noam. Can you hear me?” A pause, then the blob morphs and speaks in a deep, roboticized voice. “The real Noam is somewhere in Arizona. I was created using a vast quantity of data found online.” I study his molten form to see if I can make out a face. “I see. And are you intelligent?” I ask. “Defining intelligence is a colossal problem beyond the limits of understanding. We have to be humble. We are in a pre-Galilean stage. We don't know what we are looking for any more than Galileo did.”

The project is the brainchild of Sandra Rodriguez, a researcher at MIT's Open Documentary Lab. "Chomsky is one of the most well-documented public intellectuals," says Rodriguez, explaining why they chose the cognitive scientist and how they were able to train the system on a trove of Chomsky archives. "We wanted to create an experience that uses A.I. to discuss A.I. If we know so little about how the brain works, what exactly are we replicating with an A.I.?"

Rodriguez's piece debuted at Sundance New Frontier, where creators use cutting-edge technology to push narrative boundaries. Augmented and virtual reality have long been staples at New Frontier. But with the growing realization that VR is distinct from cinema, storytellers are embracing the medium for its unique properties and adding elements like machine learning/A.I. and biotech to enrich the experience. Even in nascent form, it's clear that A.I.-based characters allow for more powerful interactions. In a landmark day for the industry, Fable Studio's "Wolves in the Walls" won the first Emmy for a virtual being last August. The studio is behind the 8-year-old virtual companion Lucy, and the story is now available on the Oculus store. Their ultimate ambition is to build a character whose reactions are improvised but also feel natural.

But it's still early days for A.I., and most projects substitute live actors for now. Borrowing from Tender Claws’ piece, "The Under Presents," some teams are starting to use immersive theater techniques. Director Sngmoo Lee adorned his "Scarecrow" lead actor with motion capture widgets and had him hop around the viewer, interacting with them one-on-one.

"VR is not an extension of cinema. It is something entirely new," says Lee, who has experience directing traditional films. "These actors will be replaced by A.I. someday. Virtual beings are going to be the next platform."

Sensorium Studio employed a similar technique. Their entry to the festival, “Metamorphic,” is a multi-person experience that places you in a garden of swirling pastels, illustrated by Wesley Allsbrook. As you move about, colorful avatar bodies appear and weave through yours. I heard a fellow headset-wearer shriek in surprise when she felt a physical touch from one of the other avatars, subverting her expectations around which avatars were embodied and which weren’t.

"VR is not an extension of cinema. It is something entirely new."

"We continue to push into a surreal realm where real and virtual spaces commingle," says director Matthew Niederhauser. "We create moments where you might not distinguish between digital avatars and other participants, leaving you with the eerie but gripping sensation that you have lost track of who is embodied or animated.”

Despite creators pushing the medium forward, the projects still reach few viewers. Some have gone on from Sundance to reach users by distribution through the Oculus store — including “The Under Presents" (retails for $19.99), "Wolves in the Walls" (free), and “Spheres” ($9.99). But because the consumer market remains early and sales low, creators typically scrap together funding sources and grants to build the projects, often with little expectation of recouping costs. At a time when Amazon, Apple, and Netflix are in bidding wars for traditional films — Hulu just bought "Palm Springs" for a record $22m — not a single one of the 16 VR projects have been picked up from the festival this year. At prior festivals, "Spheres" and "Zikr: A Sufi Revival" were acquired. But neither of the acquirers have distributed these experiences to audiences through museums or other outlets.

Oculus (Facebook) and Vive (Valve/HTC) have sponsored VR projects in the past. This was one of the first years where they had nothing official on the slate. But a second wave of companies are now arriving on the AR side. Both Microsoft's HoloLens and the Magic Leap One collaborated on projects this year. Creator Diego Galafassi showcased a mixed reality piece, “Breathe,” on the Magic Leap. Before starting, you wrap a sensor around your chest that tracks breathing. Glowing air particles whoosh as you exhale, responding to your hands when you reach out. It's a powerful demonstration of what AR combined with biosensors might enable despite the headset's field of view — which remains distractingly small.

"Breathe" is also an example of how New Frontier content tends to fall into one of two camps. In this case, it's harnessing technology to deepen our connection with the surrounding environment and inspire wonder.

The other camp leans in the opposite direction, highlighting the dangers of technology. This year, techlash sentiment was felt across the festival. Compared with CHOMSKY_AI's broody philosophizing, “Persuasion Machines” leads viewers to a place of anxiety, pushing them to question their assumptions. The project is a spin-off of Netflix documentary The Great Hack about the weaponization of data. It feels like living inside an episode of Black Mirror. Apparently, Hillary Clinton of email hackgate would understand, as the former senator and secretary of state stopped by to try the experience, a few months after moderating a panel with the creators. "Being aware of how platforms and products use our data to target us is the first step toward addressing the dark side of technology," she would later tweet, aware of the dark side more than most. "Persuasion Machines" was, for Clinton, an "amazing VR experience."

"Persuasion Machines" lands you in a data privacy dystopia. As you walk around a bland virtual living room interacting with smart devices, you are prompted to accept their Terms and Conditions. The room progressively starts to crack and reveal the dataverse, flowing Matrix-like around you. You learn about your online footprint and why your information is as valuable as oil in the surveillance economy.

"We made the film last year and it was eye-opening. But we realized it was important to bring the conversation to life further," says director Karim Amer. "The topics in our piece are reported on in a lot of articles. But our ability to comprehend and conceptualize it on an emotional level is limited. The piece brings the everyday nature of data to life. VR takes it to another level."

Amer acknowledges that distribution platforms haven't yet been solved but hopes the team will take it to public spaces and malls. He's also aware of the irony of cautioning against big data in virtual reality, the ultimate persuasion machine. Amer considers the piece a subversive statement on the medium itself.

"We saw Facebook's irresponsible handling of data. Let’s hope Oculus doesn't make the same mistake," says Amer, noting that Facebook owns Oculus, one of the largest players in the VR industry. "We also wanted to warn VR artists that they have the potential to create the largest data source on individuals. With this unprecedented access comes responsibility. The industry needs to get ahead of that."