AI thrashes Gran Turismo pros in promising study for driverless cars

The fantasy of the driverless car has existed for more than 40 years depending on who you ask, but, despite decades of potential time for advancement and development, the technology isn’t quite there.

The AI to control fast-moving vehicles needs to be safe, timely, and respectful of the ever-changing rules of the road. But so far, results have been mixed — as Tesla’s autopilot mode can attest.

But a study published Wednesday in Nature takes an unlikely inspiration to improve self-driving AI: having the AI play video games. A team of two dozen researchers put an AI named GT Sophy into the driver’s seat of the video game Gran Turismo Sport. When it performed “wrong,” the researchers course corrected the AI until it beat some of the best human players in the world to shame.

“Simulated automobile racing is a domain that requires real-time, continuous control in an environment with highly realistic, complex physics,” the authors write. “The success of GT Sophy in this environment shows, for the first time, that it is possible to train AI agents that are better than the top human racers across a range of car and track types.”

What’s new — The team of researchers was confronting a problem: While there has been some promise in studies that leverage algorithms to predict the actions of drivers in real-world scenarios, even the smallest of miscalculations in these models can lead to physically disastrous results.

The team thus set out to leverage an AI model capable of conducting virtual races while being generally prepared for human imperfection. Researchers strove to create an AI proficient in four areas: race-car control, racing tactics, racing etiquette, and racing strategy.

The basis for the study is a type of machine learning called Deep Reinforcement Learning (deep RL). Simply put, it’s an AI construct in which the computer is rewarded for performing correct actions and punished for doing the wrong ones.

In the scope of Gran Turismo Sport, this meant being rewarded for staying on the track and making tight turns while being carefully criticized for any and all unwanted contact between competing vehicles. With this simple model, researchers hoped to create a smart AI capable of skillful video game racing.

Why it matters — If a deep RL model can be leveraged to effectively control and race virtual cars, there’s potential promise in similar techniques being used to improve AI control of physical cars.

In the near future, however, such a study is yet another landmark in proving instances where AI is capable of outperforming humans. According to researchers, this is the first known example of a study featuring head-to-head racing. By watching AI-controlled races, the game’s best esports players can become even better at their craft.

What they did — GT Sophy was put through a series of levels in Gran Turismo Sport, and rewarded for staying on the track and driving well, while also being punished for unsportsmanlike action.

When it came to learning finer details more closely tied to human behavior, the team opted to improve lap times by providing rewards for the AI when it passed another vehicle and punishments for when the AI was passed. On the contact front, the AI was penalized for any contact in which it was involved, with stiffer penalties for performance “likely considered unacceptable” per the game’s sportsmanship rules.

It took several tries, but after a few hours, the AI was capable of making it all the way around a track, and within two days, was faster than 95 percent of players. Within nine days, GT Sophy knew the game inside and out.

With the AI designed and tested, researchers’ next step involved two events held in early July and late October of 2021 against some of GT Sport’s best Japanese players. The competition involved time trials against three pros and a scored head-to-head race against four pros spread across three different tracks.

In the first time trial in July, Sophy GT emerged victorious against all three human racers, but lost in the head-to-head category by a score of 86-70. Following some small modifications, however, Sophy thrashed its human opponents in a rematch, with a final score of 104-52, this time in favor of the AI.

What’s next — While it’s clear the team behind this study has created an AI that’s extremely proficient at playing some very specific tracks in Gran Turismo Sport, there’s a long way to go for these results to have any serious real-world application.

As effective as the algorithm may be in responding to some human behaviors in a head-to-head context, it’s important to remember that this undertaking still involves one tightly controlled digital product, in a video game, interacting with another in the form of the AI. As such, it will likely take far more rigorous real-world testing to see how these principles might apply to driverless cars.

That said, as a showcase of general advancements in AI and machine learning, it’s still interesting to witness computers coming out on top in head-to-head racing scenarios.

Abstract — Many potential applications of artifcial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits1 . Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world’s best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing’s important, but under-specifed, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world’s best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defned human norms.