A Failed 'Final Fantasy' Film Brought Hollywood Around on Motion Capture

Performance capture, the art of digitally recording human movement and applying it to an animated character, has been so commonplace in blockbuster movies that now, the presence of computerized characters are almost a given. The technology is so well-integrated and accepted that today, audiences don’t blink a collective eye at fully digital pizza-scarfing turtle warriors, gigantic purple demon villains, or ferocious green monster men with soulful eyes.

That sort of fact-of-life acceptance makes it hard to believe that 15 years ago, the technology was ignored by many at traditional Hollywood studios, and seen as a risky curiosity when first presented on the big screen.

Remington Scott, now 47, fought a long, uphill battle to develop and legitimize the technology. The big breakthrough for the movie industry came when he directed the motion capture on 2001’s Final Fantasy: The Spirits Within, the first film to ever fully use motion capture, and supervised the process on the film that made it famous: The Lord of the Rings: The Two Towers.

But, as he suggests, Hollywood co-opted the process after outsiders developed it; both of those films were made by visionary directors working outside the mainstream studio system. And Scott’s journey began as far as possible from the high budget film industry: A high school on Long Island.

He was still a teenager when he and his friend, Michael Riedel, created the first video game using digitized human images. The development was a major step up from what the 8-bit gaming industry was doing in the mid-‘80s.

“We said what if we took a video tape and we digitized frames from the video and played it back sequentially?” he recalled in an interview with Inverse last week. “No one had done that as a game. When we did that, we had to literally write the software to take the images and retouch them.”

The technology was ultimately licensed by the WWF (now the WWE), and used to create the 1987 strategy wrestling game Microleague Wrestling. That led to a gig with Acclaim, the massive AAA games studio, which set up shop in neighboring Oyster Bay, NY. The studio, with Scott on board, used their digitizing technology to record people acting against blue screens and pump out a constant stream of more realistic-looking video games. The studio’s biggest seller was the Mortal Kombat franchise.

But when three-dimensional games began popping up in the mid-90s, they had a new problem on their hands: the team at Acclaim couldn’t even animate a full walk cycle, which led them to seek outside help.

“In the early ‘90s there really wasn’t a lot of software to creating realistic digital humans in 3D,” Scott said.

“They hired animators from CalArts and even Disney, and put them in front of these graphics machines and said, can you do 3D animation? And they had a hard time even starting the machines up. It was all Unix, the software was too difficult. They lasted a few weeks and went back and said good luck, we’re going back to paper and pencil.”

The tussle with animators would be a recurring theme in Scott’s career, even when the industry transitioned from traditional to the computer animation made famous by Pixar. But the more immediate problem was trying to figure out how they were going to adapt to this sudden surge in 3D gaming, which they knew was no passing fad. Once again, the habit of searching outside the industry for possible solutions was key.

“They looked to the medical industry and the military industrial complex, and they found this technology that was used in labs in the medical industry,” Scott said. “They put these giant ping pong balls on legs and hips and guys would walk on a treadmill and be surrounded by a small array of data that would record this information. What it was looking for was beneath the skin and muscle, and determined how the ball of the femur was rotating in the hip, for hip replacement surgery. So we saw if you could bio-mechanically recreate that info, you could create a full body, and then we could scale up and create multiple people.”

There was still the tricky task of converting hard data into visual animation, which required Scott and co. to once again develop an entire suite of software tools. But once they got going, Acclaim was back on track; the technology was used to make top-selling titles, including the Turok: Dinosaur Hunter, NFL Quarterback Club and All-Star Baseball series in the mid-to-late ‘90s.

VFX houses adjacent to Hollywood were rapidly developing new digital technologies, using CGI to create animals and wild stunts and effects that would have been impossible to pull off on a real film set; Jumanji, with its stampeding horde of jungle predators, was a great early example of the success they would have with this new tech. But though studios knew about motion capture, Scott says they wouldn’t yet commit to actually using it, since it wasn’t a proven commodity for movie production no matter how hard he tried to push it.

“At the time we were doing this in the ‘90s, we saw Jurassic Park, and we were like, Wow, look at what Hollywood is doing with computer generated images,” Scott recalled. “These dinosaurs looked real. We thought, if we can work with Hollywood and take realistic human motion and use their render tech, we could make some realistic looking movies. We went and pitched studios and produces and VFX supervisors. And across the board, everyone in Hollywood, they said ‘Why would we make digital humans in 3D when we can shoot them on the green screen?’

Other than Jar Jar Binks in Star Wars: The Phantom Menace, who was greeted with unanimous derision, there had been no motion capture character to make it to screen in a major release. Even as other technologies took off in Hollywood, mo-cap looked like a lost cause — until Scott got a call from Japan.

Here was the pitch: Hironobu Sakaguchi, the creator of the smash hit Final Fantasy video game series, “wanted to make an animated movie that wasn’t like an animated film, Scott recalled being told. “He didn’t want it to be a cartoon like everyone else was creating. He wanted it to have real emotional characters that were physically real and looked as real as possible.”

So he flew out to Hawaii, where the film had been in production for over a year, with little to show for it. The same battle he faced at Acclaim was being waged at the island set, where Square Pictures was trying to create the most lifelike animation ever seen on film, but struggling with both technology and reluctant animators.

“They brought on an American team to help them animate Final Fantasy, and the animators didn’t even want to try,” Scott said. “[The production] was trying to shoot stuff with motion capture, but when they would come back in, animators would go and animate it instead. The director said it didnt look like humans.”

Basically, the animators weren’t all that interested in using the data that was being delivered to them after the motion-capture shoots, and Sakaguchi was growing frustrated. And so, after a year of work, the animation team was let go and Scott came aboard. He helped write some code that improved the look of the motion capture, and a new animation team more willing to work with the new tech was hired.

After a character redesign — the originals “looked like Barbie dolls,” requiring a lot of late night work by a designer named Steve Giesler — Sakaguchi and his team got to work on making history.

“Motion capture was used for all the bodies of all the characters,” Scott said. “The faces had very complex facial rigs that were being keyframed. So it gave animators the ability to take the body motion and ADR session and then keyframe the faces.”

The result was stunningly realistic movement being performed by computer generated characters. They could have even taken it a step further, but there again Scott faced the pesky reality of reticent animators worried about losing their place in the industry.

“We did tests where we recorded facial motions and we solved it on to the face, and it looked really good, but you’ve got to understand that opened up a whole can of worms,” he said. “People were like, ‘if you’re gonna do the body and faces, and what do you need animators for?’ So there were compromises in that field. We have an incredible animation team and they wanted to do those faces.”

The animators certainly delivered, creating hyper-detailed faces, down to the smallest tics and breathing patterns. They were almost human, certainly more than any other CGI creation in history. And while the movie was technically being made outside of Hollywood, it did not lack for attention and hype.

There were articles in national publications such as Time and several profiles in the New York Times focused on the spectre of the digital actor one day replacing the real thing. The lead of Final Fantasy was a character named Aki Ross, and Square Pictures had big plans for her, beginning with bikini photo shoots in Maxim (she made the magazine’s 2001 Hot List). She was voiced by Ming-Na Wen and physically performed by an actress and martial artist named Tori Eldridge, but Ross belonged to Square, and the studio intended on putting her to work. There were plans to utilize her flawless body in future movies and other media. Driving home her viability as a new kind of 21st century celebrity was a CNN segment in which a reporter showed her photo to men on the street, many of whom said they’d love to date the digital character.

Despite the hype, the movie wound up earning lukewarm reviews (though Roger Ebert loved it) and bombing at the box office, taking in just $85 million worldwide on a bloated $137 million budget. While it was not much consolation for the higher-ups at Square, which was ultimately forced to take a capital injection from Sony and then merge with the Enix video game company, Hollywood took notice of what they had created.

The Wachowskis, who had just done their own part to push the introduce digital innovation into the industry with The Matrix, tapped Scott and Square to handle the motion capture on their short film The Final Flight of the Osiris. At that point Square was shutting down its movie studio, but Scott and a few colleagues agreed to hang around long enough to work on the short, which served as a segue to the second and third Matrix movies.

Things snowballed from there; the next person to call was Peter Jackson, who was having trouble with nailing the look of Gollum in his second Lord of the Rings film.

Initially, Andy Serkis had been hired to simply provide the voice for the cave-dwelling ring coveter, as they intended on fully animating the character. But Serkis had to be on set to read lines with actors, for the sake of timing and more natural reactions. His presence on the soundstage would change everything.

“When they were shooting them, a strange thing would happen: Everyone would turn around and look at Andy, because he was rolling on the ground, crawling around, physically getting into it,” Scott said.

“They were like, ‘Look at this guy, it’s amazing.’ And so PJ said, let’s put him in front of the camera and do a take where we get this and use it as an animation reference. So they let him do one take, with the actors. The other guys would get as many takes as they wanted, but that one take with Andy was usually the best, because of the energy that everyone was having together. And PJ saw that, and he knew that he had to get Andy’s performance.”

The same old problem presented itself: The animators weren’t using Serkis’s actual movements to shape the character, but instead just continuing to animate him the way they thought Gollum should look. So again, Scott was called in, and the rest is history: Lord of the Rings: The Two Towers won an Oscar for visual effects — its motion capture was even more advanced, with real-time preview rendering during shoots and the industry rushed to adapt motion capture, which soon became known as performance capture.

Robert Zemeckis, who used digital image manipulation to great effect in Forrest Gump, made the mocap holiday movie The Polar Express with Tom Hanks, which digitized the Oscar-winning actor’s face, another step for the technology. Scott went to work at Sony, where he helped direct the motion capture on Spider-Man 2 and Spider-Man 3, Superman Returns, and Watchmen, among other titles. And the technology continued to get better, by leaps and bounds.

“What’s happening now is that you’re having a performance created completely unified,” he explained. “You’re recording your actor’s body, face, voice, everything simultaneously right now. Hollywood didn’t understand it or trust it, and it was not something you could get an A-list actor to come into a motion capture stage to do.”

That’s no longer the case; Scott directed Oscar-winner Kevin Spacey in the stunningly realistic Call of Duty: Advanced Warfare, The Hulk looks almost disconcertingly like Mark Ruffalo even when swelled up to the size of a Midtown office building, and Andy Serkis has been so successful at bringing his characters to life that people have pushed to make performance capture artists Oscar-eligible. The recognition hasn’t yet been granted, but progress is inevitable — that much has already been proven.