The FBI’s Massive Facial Recognition A.I. is Based on Bad Science

There’s a fifty percent chance that your face is in a government criminal database used by the FBI in investigations. For law-abiding citizens, that might not seem like a huge deal, but on March 22, third-party investigators said the entire system was reckless, unproven, and biased during a House Oversight Committee hearing.

In the hearing, data analysts at Georgetown Law School, advocates at the Electronic Frontier Foundation, members of the Government Accountability Office, and other groups pointed out clear problems with the database.

First, even the FBI’s own tests show the system is only somewhat accurate. When the system was asked to pull the 50 closest-matching faces from a set of nearly 1 million, it got the right one only 86% of the time, according to a 2016 GAO report.

What’s more worrisome is how many tests the FBI hasn’t done. The Bureau has not tested the accuracy rate when pulling candidate lists of fewer than 50 potential matches — a problem given that system users can request a list with as few as two potential matches. “In the real world … users only generate the top handful of images which requires a much higher degree of accuracy,” GAO’s Diana Mauer said in the hearing.

And while the tests it did focused on false negatives, to check that the system didn’t ignore a match, the Bureau hasn’t done any false positive testing to see how often non-matching faces are flagged as potential hits.

That’s not an acceptable practice in the field.

“According to the National and Science and Technology Council and the [NIST], the detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition,” the GAO report said.

As Georgetown Privacy tweeted: “False-positive testing is the bedrock of #facerecognition technology. That’s why we recommended the @FBI do it.”

Then there’s the problem of biased results. A 2012 study co-authored by the FBI showed that the system was significantly worse at identifying African-Americans, women, and young people. Despite knowing about this weakness, the Bureau doesn’t account for varying accuracy by race.

“Our requirement when we developed the interstate photo system did not depend on tone or ethnicity,” FBI Deputy Assistant Director Kimberly Del Greco said in the hearing. “It was based on the mathematical composition only.”

Another criticism raised at the hearing was that African-Americans are reportedly more likely to be targeted by police facial recognition.

“According to these reports, if you’re black, you’re more likely to be subject to this technology. And the technology is more likely to be wrong. That’s a hell of a combination,” Maryland representative Elijah Cummings said in the hearing.

Altogether, these alleged missteps on the already controversial program have experts worried.

“This is unproven technology,” Jonathan Frankle, a computer scientist who worked at the Center on Privacy and Technology at Georgetown Law School, tells Inverse. “If the FBI is using a system and they don’t understand the technical characteristics of that system, it’s reckless … It’s engineering negligence.”

Case in point, Frankle says, is the failure to investigate why facial recognition was worse for African Americans.

“I wish I had the answer for you but no one has done the research,” Frankle says. “It’s an embarrassment on me and my community that the last major study on this was done in 2012.”

Experts at the hearing also criticized legal missteps in the construction of the facial database. Most notably, the FBI had failed to submit a Privacy Impact Assessment to the Department of Justice as was required, both when it started using the system in investigations and when collecting driver’s license photos by the millions.

Despite the system’s flaws, Frankle says he’s not opposed to the technology when implemented properly.

“I completely believe [the FBI] when they say it’s a powerful policing tool,” Frankle says. “You should use them sparingly because they can be invasive, and you should test them regularly to know how they fail. That’s just good engineering.”

You can watch the entire House Oversight Committee hearing below: