Brain Training: A Case Study in Scientific Spin

The brain training industry has come under fire a lot in recent years for over-hyped claims that playing skill-based computer games can make you generally smarter and protect you from losing your mind as you get older. Earlier this year the company behind Lumosity was fined $2 million after the Federal Trade Commission found its advertisements promising improved neural function to be unsubstantiated by fact. In 2014, dozens of psychologists and neuroscientists banded to express frustration, worry, and caution regarding the burgeoning industry, citing a lack of evidence that brain training games reduce the risk of dementia or Alzheimer’s disease.

With all that baggage, it’s all the more shocking to read the headlines after a research presentation at the American Psychological Association’s Annual Convention last month. “Play on! In a first, brain training cuts risk of dementia 10 years later,” reads the headline in Stat. “‘Brain training’ cut dementia risk in healthy adults - U.S. study,” exalted Reuters.

But let’s all take a step back here: These jubilant headlines are based on a study that has not been published or subjected to peer review.

To call the findings preliminary barely scrapes the surface. Admittedly, this research could become the first large-scale randomized trial to link brain training with long-term dementia risk — but that should be a signal for caution, not celebration, an encouraging sign but not necessarily a definitive one. Yet BrainHQ, the company that sells the specific training game involved in the research, is peddling the science as settled. “A unique brain exercise cut the long-term risk of dementia nearly in half in a large study of older adults,” according to a news release.

There are lots of real and potential methodological issues with this work, and Hilda Bastian goes into detail on some of them here. But here’s the biggest, most glaring, staring-in-your-face problem: Those people who saw a 48 percent reduction in dementia risk compared to the control group over ten years? They weren’t random. Only those who were selected for a “booster” of additional training after the first round showed this result, and in order to qualify, you had to have shown up to at least eight of the first ten training sessions.

This is from a 2014 research paper based on the same data from the same 10-year research study:

“We note that the evaluation of the effect of booster training is limited because the two groups of interest (booster trained and non-booster trained) are not comparable. In order to be eligible for selection for booster training, participants had to have completed at least 80% of baseline training. In contrast, only 20% of non-booster trained participants completed baseline training. Therefore, the non-booster trained group is overrepresented by persons who did not complete baseline training, and reflects neither participants who completed baseline training nor non-trained participants (i.e., the control group) but something in between.”

Suddenly, boasting of your large-scale, randomized trial seems somewhat less impressive. And that 2014 article? Participants who got training said they improved at the tasks of daily life, although researchers were unable to show a significant and measurable improvement.

This is not to say that brain training games are useless. The specific technique in question, called “speed of processing” training and marketed by BrainHQ as “Double Decision,” has shown to have spillover effects into the lives of people who use it, including improving their driving skills. That can be a big deal for an older person trying to hold onto their independence.

But there are many steps to go before the company can solidly back up a claim of reduced dementia risk, and getting this research reviewed and published is just the first.

“A single study, conducted by researchers with financial interests in the product, or one quote from a scientist advocating the product, is not enough to assume that a game has been rigorously examined,” write the scientists in their consensus position on the Brain Training industry. “Findings need to be replicated at multiple sites, based on studies conducted by independent researchers who are funded by independent sources.”

In this case, the lead researcher has worked as a paid consultant to the company that owns “Double Decision,” but declared no interest in the future success of the product.

Hopefully you’re not at a stage in life where you’re worried about mental decline, but here’s a totally unproven way you can be sure to stay healthy and sharp until the day you die: Every time you see a claim that a commercial product is “Proven, By Science!” dig a little deeper. Developing critical thinking skills is guaranteed to also make you happier and better at sports, because brains are weird and who really knows how they work, anyway?