Here's How Google Made VR History and Got Its First Oscar Nom

The team at Google Spotlight Stories made history on Wednesday, as its short film Pearl became the first virtual reality project to be nominated for an Academy Award. But instead of serving as a capstone, the Oscar nod is just a nice moment at the beginning of the Spotlight team’s plan for the future of storytelling in the digital age.

The Road to Spotlight Stories

Google Spotlight Stories are not exactly short films. Rather, they are interactive experiences created by the technical pioneers at Google’s Advanced Technologies and Projects (ATAP) division, and they defy expectations and conventions. Film production has in many ways been perfected, but for each Spotlight Story, the technical staff at Google uncovers new challenges to telling stories in a medium that blends together film, mobile phones, games, and virtual reality. Needless to say, it’s been an interesting road.

Google’s (ATAP) division is an “incubator of projects where Research & Development finds interesting applications in a specific domain,” Rachid El Guerrab, technical program lead at ATAP, told Inverse. The interest in telling stories on a mobile device came about while ATAP was still a part of Motorola: “The mandate was: Can you make emotional content for a phone?” El Guerrab recalled. Spurred by questions about how people are connected to their mobile phones and what they do with them, the goal was to deepen that connection in a meaningful, organic way. The solution to connecting with new technology: man’s oldest trick, storytelling.

Now, the state of mobile phone technology is able provide the platform necessary for the team’s ambition. “The combination of motion sensors and the fact that these phones now have the graphics power of some high end consoles meant that we could turn that display screen into a window into virtual art,” said El Guerrab. “So we started forming this idea of making a story or a movie, but giving the camera to the viewer. That was the first moment it crystallized into new, interesting ways of telling stories.”

As the history of film has proven, any time a new technology is introduced into the creative process, there’s an adjustment period while the artists and technicians figure out the limitations and the possibilities. “We had just lost a hundred years of learning about framing and composition and montage,” El Guerrab conceded. “So how do you actually tell stories and still be able to get as much impact as you’re getting from film?”

Guiding the Viewer

The first challenge that presents itself when you relinquish control of the camera to the viewer is how to deal with the fact that they can look anywhere. Since the experience of watching one of these shorts requires a certain amount of physical movement on the viewer’s part to ensure that the action remains inside the frame of the phone’s display, considerations from the world of theater have to be taken into account.

But you can’t always guarantee that your viewer will look where you want. “Handling of your attention as a viewer has been the biggest challenge and the one where we put the most effort in figuring out what works,” said El Guerrab. So what happens when the viewer doesn’t look at the right place?

Their initial tests were simple: A viewer had to follow a pig running around a forest. At certain points, the pig would perform an action, or story moment. One of the first solutions was something called a “follow camera,” which assisted and guided the viewer. As long as they held the phone without moving, the camera would frame the pig within the display. The moment they moved around, it no longer followed the pig. The idea that resulted was that the story would wait for the viewer until you reframed the camera on the main focal point of the narrative.

On their first released short, Windy Day, a story about a mouse whose hat gets caught up in the wind, this concept was pushed even further, in order to hide cuts and changes in setting. “If I’m following the mouse, everything behind me could change completely,” El Guerrab said. “Though it’s similar to a cut, it’s more of a ‘spatial cut’ in that it’s based on where you are looking and when.” By changing the set in ways that are invisible to the viewer — i.e. not happening directly in front of the camera — it granted Windy Day a sort of narrative flow, which could take the viewer from inside of a tree, to inside a tornado, etc. without being jarred by an obvious cut or jump in the story.

Another key lesson the Spotlight Stories team learned about guiding the viewer’s attention came from animation veteran Glen Keane (director of The Little Mermaid). On Duet, an emotional hand-drawn short, the performance of the two characters evoked a clear sense of intent. For example, the boy climbs a tree and he looks in one direction, suggesting to the viewer that they look in that direction as well. Having the characters provide hints about where the viewer should look created an organic way for viewers to navigate through the narrative space.

Writing a Story for a Space, Not a Screen

The initial development period for each of the Google Spotlight stories begins with questions about how the concept of space affects the story and the viewer’s interaction. “Before we even go into a project, we always ask, ‘Why is this a story that needs to be told in a immersive way?’ If there is no reason for that, then why not just tell the story in a 2D frame?” El Guerrab said.

From the front seat of the car in Pearl to the town square in Rain or Shine to the Rear Window effect in Special Delivery, there is always some statement about the importance of space in relation to the story and the viewer.

Since there aren’t any concrete rules for how people watch short films on their phones, there is a conscious effort at the early concept phase to account for active viewers vs. passive viewers in a narrative space. Questions of user engagement and interactivity affect the development of not just the main narrative thread, but also secondary story moments that a viewer can stumble upon.

Nexus, the animation production company that made the short Rain or Shine, created a 360 degree script for the town square. In the narrative matrix they created, something is happening elsewhere in the town square that prompts the viewer to look for the little girl again. For instance, a cat chases a bird and their path eventually guides you back to the main story. Even if you’re actively wandering around the space and discovering these little side stories, the idea is that they tend to bring you back into the main linear narrative.

The process for developing the stories in a space isn’t standardized, however. For Duet, there was a script with a series of sticky notes to figure out the intertwining storylines. On Pearl, director Patrick Osborne, who won an Oscar with Disney, started with ordinary storyboards to nail down the emotional arc and then transitioned to a series of exaggerated forced perspective layouts to simulate what a viewer would see through the phone. On Help, the big discovery came during previews, during which director Justin Lin found that the pacing of the narrative is directly correlated to the time it takes for a viewer to physically move the camera frame to a relevant story moment — it’s a big change from the rapid pacing he was accustomed to on the Fast and the Furious franchise.

With each new Spotlight Story, new discoveries about the interaction between story, space, and technology bubble to the surface. “It’s like architecture,” said El Guerrab. “When you are designing a physical space, you don’t know where people are going to sit, where they are going to look, what angle are they going to see. You are trying to create the whole environment to put them in a certain feeling, a certain of understanding of space.”

Pearl: Rewarding Narrative through World-building

With over 15 years as an animator on films such as DreamWorks Animation’s Shrek 2 and How to Train Your Dragon, Cassidy Curtis has picked up more than a few tricks about storytelling. As the technical art lead on the Spotlight Stories team, Curtis has put his experience in film into solving the many challenges presented by this new medium. Yet, despite his broad scope of responsibilities (which can range from creating custom production pipelines to figuring out how to create precise color controls down to the pixel level), it is all in service of guiding the viewer through a world.

On the Oscar-nominated Pearl, Curtis put his experience to the test by translating the illustrative qualities of production designer Tuna Bora’s artwork into a cohesive 3D environment, all while making sure it ran smoothly on a mobile phone. As an added challenge, they had to figure out how to incorporate the aesthetic details into every nook and cranny of the car that might tip the viewer about narrative clues.

“In film, every single member of your audience is confronted with the same rectangle,” Curtis said. “You can predict with amazing accuracy where people’s eyeballs are going to be pointed at any given moment. When you are thinking in terms of 360 video or VR, you have to think of the audience as a spectrum of people — some will look here, some there. There is never a moment where you can say with absolute certainty, ‘The audience does this.’”

Pearl finds a way to make the discovery of garbage and household objects meaningful. Shoes, hamburger wrappers, frying pans, or laundry soap placed on the floor of a car in which the viewer “sits” become significant clues to the story. By incorporating these small details in the set dressing, the Pearl team created an added level of insight into the characters’ lives. This type of reward for exploration is another tool in the storytelling arsenal that has developed with each subsequent Spotlight Story and one that relies on the viewer to make the narrative leaps that fill in the backstory for these characters.

Changing Animation

Led by Osborne, the crew of animation veterans on Pearl came from Disney, PDI, DreamWorks, and Pixar, which meant they had the best pedigree possible. And yet, new, unexpected challenges often cropped up during the production of the short. Given the short’s unique flat-shaded visual aesthetic — which was developed by production designer Tuna Bora — there was a need to have a very naturalistic style of movement. This required a new way of thinking when it came to reviewing work in the new medium.

“Animators have to change the way they think of animation, because now theyre not animating to a frame or to a specific angle,” El Guerrab said. All told, the production schedule for animation was fairly short, around two months, but because the rules of the medium were still being figured out, reviewing work in a way that took into account the viewer’s changing perspective proved to be a challenge.

“We can’t have everybody put on a headset or have everybody looking at a separate phone [in dailies],” Curtis said. “So, the animator would just shoot the scene from one particular camera or maybe two or three different cameras, then we would review that the same way you would review a shot in an ordinary animated film.”

They ran into trouble at times when they “would forget to look down and see what the characters’ feet would look like,” said Curtis. “In particular, there was one shot where the dad gets into the car and angrily starts the engine and drives away. And we didn’t notice until the animators were long gone that his feet weren’t actually touching the gas pedal. And so then somebody had to go back in and animate his feet pressing the gas pedal down.”

Why Formats Matter to Form

“While we were building the story for Pearl, Patrick always had his kind of ideal path through the story,” Curtis said. The notion of a viewer who latches onto the characters and follows their emotional arc to the exact framing the director originally intended is something quite difficult to achieve, when all control over the camera is handed over to the viewer.

Surprisingly, however, it wasn’t as difficult to achieve in virtual reality. “You can respond to curiosity with less energy in full VR than in the handheld version,” Curtis said. As a result, the majority of viewers will actually follow exactly the path that the director hoped they would follow. “Of course, there are always those VR enthusiasts who will push their head through the top of the car just to explore, but for the most part, almost everybody looked exactly where we were hoping.”

Designing for emotional engagement was at the very core of the initial mandate for the Spotlight Stories team, so now that VR has entered into the picture, there is an added component of shaping the experience to the new modalities. “The biggest changes [to Pearl] were actually the camera and the characters,” Curtis said. “In the initial pass, the camera is in between the driver and the passenger. But, if you are in VR, the immersiveness of it makes it feel like the characters are right in your face. It’s uncomfortable. So we actually killed characters in certain scenes or we moved them into the back seat so that you could sit in the passenger seat.”

In addition to VR, there are four other formats into which the Spotlight Stories team is adapting their content. First, there’s the 360 linear video, which is your standard YouTube 360 video, which features no interaction. Secondly, there’s the 360 interactive video, which is the version featured on the App. This version has interactive elements, and the story waits for you, but you have to hold it at a distance from your head. “We use zoom for the camera to add some parallax to some scenes,” El Guerrab said.

Third, there’s the smartphone-based VR (such as Google Cardboard), which is essentially the same as the 360 interactive version but strapped right onto your face. This changes the experience by bringing you into the scene and eliminating things in your peripheral vision. Fourth, there’s the VR version with six degrees of freedom, where the viewer is allowed positional movement within the virtual space. Lastly, there’s your standard run of the mill 2D version. Pearl was the first Spotlight Story to be done in all five formats, and they each came their own design challenges.

The Future of Spotlight Stories

To date, the team at Google Spotlight Stories and ATAP have partnered with filmmakers, animators, and studios from all over the world and from a variety of disciplines. The Spotlight Stories app has featured traditional 2D animation, live-action VFX, and CG creations, each with its own unique flavor and emotional hook.

The future of these partnerships, both with individuals and studios, stems from the Spotlight Stories “Story Development Kit,” or SDK, which gives creatives the tools necessary to dive in and simply create. “We gave [the SDK] to them and said, ‘You guys make something beautiful and call us if you have any trouble.’” As more Spotlight Stories test the SDK, more problems will be solved and we’ll see more inventive uses of the technology to create these small pieces of groundbreaking storytelling.

“Our main mission is to enable as many places, as many studios or teams or enthusiasts as possible,” said El Guerrab. “To be able to tell stories without the overhead. We are trying very hard to keep it very close to a film pipeline, all the way from conceptual to the end. It’s all a giant experiment so far.”