The Paradox of Safety Testing Autonomous Cars

A motorcycle helmet strapped onto my head, I nod at my driver as he charges into the beginning of a hot lap, snaking through hairpin turns in a black Dodge Charger, the wheel screaming and spinning out behind us. This is one of the final tests for passenger cars, to determine how the vehicle will handle under the extreme condition of sharp turns on uncertain roads. I laugh almost hysterically as we hit gravel purposely thrown on the track and the back end of the car whips out more dramatically.

But despite the chaos, my driver laughs. “What seems to be losing control and causing the rear end to slide,” Mark Tami says, “is actually being able to make it slide where you want to and make it stop sliding when you want to.”

Tami was then the director of the Transportation Research Center, who left the position shortly after I visited in late June. I had traveled 40 miles west of Columbus, Ohio, to the TRC, the largest independent vehicle-testing center in North America, to understand how engineers at the center are starting to design safety tests for autonomous cars.

Looking down the winding road out of the test site, Tami sobers up a bit as we drive out toward where an autonomous car testing site will soon be built. As he explains, a test driver has to imitate losing control to see how the vehicle reacts, but that becomes a paradox when it comes to testing autonomous cars.

To find a car’s limits, engineers push test vehicles to ridiculous extremes. But the algorithms that drive autonomous car are designed specifically to avoid these scenarios in order to keep drivers safe. This leaves a massive challenge for automotive engineers — how do you push autonomous cars to the extremes you need for testing scenarios when the algorithm is fighting to prevent those dangerous events from happening?

Imagining the car as the driver has led to new approaches in testing and an entirely new testing facility, but the end point remains elusive.

“Autonomous cars have captured our imagination, and we control our own destiny. So yeah, they’re coming,” Josh Every, Automated Vehicle Lead and Interim Manager of Research & Engineering at the TRC tells me with a laugh.

Autonomous cars have already forced Every to change how he thinks about automotive engineering. “One of the first exercises I did was every day getting in the car, turning on my GPS, which adjusts to the traffic conditions, and doing exactly what it told me. Even if I didn’t believe it, even if I thought it was the wrong move,” he says with a laugh. Even with a two hour commute he’s rarely had problems. “You have to release control to that level to start to understand how this is going to change life.”

As we drive through the 4,500-acre testing facility at the TRC, Tami points out a rough road test that looks like a concrete mogul course to me. A driver has to go over the moguls at a high speed over and over to test durability, says Tami. That requires a kind of control — or the imitation of the loss of control — that’s difficult to test on an autonomous car. “We want to be able to simulate having the vehicle, who is the driver, simulate what would happen with the real [input] and then simulate failure modes where the driver creates the wrong input,” says Tami, shaking his head.

But an algorithm that offers intentional wrong reactions like you would need for sliding doesn’t tell you anything about the quality of the self-driving system. This means Every and his fellow automotive engineers have to get creative. “An automated vehicle, for testing purposes, is a black box that converts an environment into an action,” says Every. “It’s my responsibility to present it with an environment and see what the vehicle does.”

A typical testing scenario for a standard car today has a highly skilled driver take a car through pre-set courses, like the various mud pits and brake slopes that are sprinkled across the grounds at the TRC. Right now, with current cars that are partially autonomous, Every has to work to design tests of things like lane-departure systems and automatic braking, as well as self-driving systems that can manage under driver supervision. On a normal day, he runs tests to uncover a specific problem that’s been discovered in an algorithm — maybe something like a shadow falling on a worn-down road — and the algorithm becomes unable to recognize the edge lines. He then has to build that environment exactly so the car can run through it a hundred times to uncover the issue.

“The main difficulty [for autonomous cars] is that the range of potentially unsafe conditions that these vehicles could encounter in normal driving is vast, and the crashes normally occur when encountering combinations of rare events, which are very rare,” Steven Shladover, Program Manager for California’s Partners for Advanced Transportation Technology tells Inverse. To help try and build combinations of these edge cases, the autonomous testing facility at the TRC will have four main testing facilities when it’s completed over the next few years, Tami says.

The facility will transform 540 acres of unremarkable scrubby grass in the middle of the TRC that Tami points out as we drive past. For turning issues, the TRC is installing a concrete platform that’s 3,000 by 600 feet, roughly the size of 10 football fields covered by 50 lanes of traffic, that can be altered to fit any problem.

There will be an intersection that’s six lanes by six lanes in which semi trucks enter at 45 miles an hour. There will also be 120 acres of city blocks and a wooded road to test signal strength in rural areas. “If you’re developing autonomous vehicles you want all kinds of [proving grounds] and lots of them to test potentially dangerous situations,” says Tami.

Every agrees that flexibility is all-important. “You might have an algorithm that was designed to get to the limits of the tires, or one that was designed to get you to the local grocery, and those two cars will handle a situation entirely differently.”

We are sitting in the conference room at the TRC, and I can look out and see corrugated steel mechanical bays that could hide prototype cars or testing scenarios. A lot of the issues Every sees in the partially autonomous systems we have today require personalized testing scenarios to figure out. It’s something that he finds fascinating, but his voice and demeanor droop when I ask about how he decides he’s done enough testing.

Despite regularly pushing the limits of testing, figuring out how to get autonomous systems to be as safe as possible is something that faces Every every day as he drives, he says. “I’m part of the community I’m trying to affect. And every person I pass on the roadway is one of my customers whether or not they know it,” he says. “One of the things I’m really trying to focus on is how much testing is enough, and I’m really not sure there is enough.”