This A.I. Technology Could Kill Off Hollywood's Green Screens

In theaters, Spider-Man swings over the streets of New York, gliding across impossible gaps, somersaulting through explosions. In the studio, however, Tom Holland rolls around in front of a giant green sheet. But fortunately for many actors, a new A.I. technology could bring an end to the tedious experience of acting in front of a green screen. A new paper co-published by Adobe, the University of Illinois, and the Beckman Institute for Advanced Science and Technology explains how a new A.I. program could remove the background behind a person or object in virtually any situation, letting actors do their jobs in slightly-more-natural circumstances. The technique could be the beginning of the end for the venerable green-screen (or “chroma keying”) technique, which has been putting actors on Mars and inside of volcanoes for over eighty years.

The problem of background removal is trivially easy, so long as there’s only a very narrow range of color you want to remove. With a green screen or other monotone set, the background can be made transparent by filling everything in the frame other than the subject with an uncommon color like bright green or blue. Once it’s gone, another background can be put in its place. But without that easily identifiable color, how is a computer to know what to remove? Check out the visual effects compositing from Game of Thrones, and you’ll see green screens all over everywhere.

So, filmmakers need a tool that can figure out which elements of an image are “background,” without the background having to be a prepared in advance. Traditionally, this has meant dividing the image into a “tri-map” of foreground, unknown, and background, and then applying the transparency accordingly. Prior methods looked for things like the edges of foreground objects and the level of blur in average backgrounds.

The video below shows how even the green screen approach was often not forgiving enough for low-budget amateur productions.

The innovation here, as in so many modern breakthroughs, was to shift the approach from classical algorithms to ones based on machine learning. In particular, deep learning neural networks were able to look at an enormous training dataset of almost 50,000 images, and then apply that knowledge to 1000 testing images. The A.I. would guess at the foreground and background elements of the training images, then compare its guesses to the known reality for those images, and adjust its method accordingly.

The result seems to be an A.I. approach that can rival the best in background removal using green screen sets, but without the need for the green screen set, itself. Unlike prior approaches, it can also be used on images with real outdoor backgrounds and all sorts of lighting conditions on the foreground object.

That could dramatically reduce the cost of special effects shots, especially outdoors, and quite simply revolutionize the visual authenticity of the burgeoning genre of green screen YouTubers. Those three-fingered hands and half-disappearing jaws could soon be a thing of the past.