World Cup Holograms: How Augmented Reality Will Change How We Enjoy Soccer
This is like something out of Star Wars.
by Danny PaezThe first few opening matches of the 2018 FIFA World Cup have already mesmerized soccer fans across the globe. But as millions gather around their TVs and smartphones to catch the action on the pitch, University of Washington researchers have already developed the technology that enable fans to watch immersive holograms of matches instead. Rather than squinting at a screen, you’ll soon be able to watch life-like miniaturized Messis and Ronaldos duke it out on your kitchen table.
That’s according to four computer scientists who say that they have used artificial intelligence to transform YouTube videos of soccer games into fully interactive holograms, according to a paper that will be presented at Conference on Computer Vision and Pattern Recognition in Utah on Monday.
“Our goal is to make the experience of watching soccer more immersive,” first-author Konstantinos Rematas tells Inverse. “Instead of watching a flat 2D screen, we want to bring you near the action. We achieve this with augmented reality. A soccer game is represented as a 3D hologram that can be placed for example on your coffee table, allowing the user to observe the game from different viewpoints, move around, and enlarge the players.”
From a YouTube video frame, the researchers recover camera parameters using the field lines. They then extract bounding boxes, poses, and trajectories (across multiple frames) to segment the players. Using a neural network trained on video game data, they reconstruct per-player depth maps on the playing field, which are rendered in a 3D viewer or on an AR device.
Rematas and his colleagues were able to develop this sci-fi viewing experience using Electronic Arts’ soccer simulator, FIFA, and Microsoft’s HoloLens.
Researchers used the video game to train a neural network to learn how to pinpoint players on the pitch. The A.I. was then able to take what it learned from analyzing 12,000 player models and turn prerecorded matches into holograms that viewers could enjoy by slipping on a pair Hololenses.
The dream here is to one day be able to create holograms of matches as they unfold live, but Rematas also says that this type of technology could one day be used to present individualized soccer statistics for each player on the pitch. That would make soccer more suitable to other gamified elements like fantasy sports.
Starting from YouTube frames (top row), the depth maps reconstructed by the neural network ca populate a virtual 3D soccer environment, shown here as mesh-only and textured renderings (rows 2-4).
“Our current implementation is not real time, but we are working on a version that could potentially allow live processing,” he says. “Getting [statstical] annotations will definitely enhance the viewing experience and could be easily incorporated into our system. We already estimate player information such as 3D position and trajectory. Therefore we can associate external annotations to the 3D players and visualize them together with the original hologram.”
Besides learning how to pull off live data processing, Rematas also said that there will need to be some improvements made to current AR headsets in terms of battery life and processing power before you can watch matches on your tabletop. Whether these devices will be ready in time for the next World Cup remains to be seen.
But the gaming and sporting implications of being able to study soccer matches with the precision of Count Dooku pouring over war holograms seem too great for the technology to remain in the realm of science fiction for long.
