Massive review shows “science” behind most mental-health apps is wildly flawed

The Covid-19 pandemic has taken a toll on the mental health of countless Americans. To make matters worse, necessary stay-at-home measures hamper access to care. In the gap, some people have turned to mental-health apps for solace: Some of these claim to provide everything from cognitive behavioral therapy to guided meditation. But a new study published in PLOS Digital Health finds these apps are not backed by the rigorous evidence their claims require — they also just don’t seem to work all that well.

What’s New — The meta-review examined 145 studies on phone-based mental-health interventions. Taken together, the authors say the methods used to study these apps are flawed. The research “failed to find convincing evidence in support of any mobile phone-based intervention on any outcome,” the report states.

“Across the literature, we saw a general pattern of weakening evidence… and diminishing effect sizes as the comparison condition became more rigorous.”

Generally, the apps were more effective than no treatment at all. But their usefulness waned when they were compared to other treatments or interventions for mental health. This was true regardless of whether the apps were targeting anxiety, stress, feelings of depression, tobacco or alcohol addiction, or general wellbeing.

“If you ask me my personal opinion, from reading the literature, I think there’s fairly strong evidence, actually, that these apps on average offer small-to-moderate benefit relative to nothing,” Simon Goldberg, an assistant professor in the Department of Counseling Psychology at the University of Wisconsin-Madison, tells Inverse.

Why It Matters — Mental health apps are a hot market. There are between 10,000 and 20,000 such apps available to download on your smartphone, according to one expert opinion. These include everything from mood diaries to programs to quit smoking, to one-on-one text messaging with a licensed clinical therapist.

This is big business: Ads for Talkspace and BetterHelp are unavoidable, particularly if you are into podcasts; BetterHelp was the largest advertiser on podcasts in 2021. Worldwide, people are expected to spend $500 million on mental health apps in 2022. Yet these apps have varying degrees of scientific backing and there’s little out there to help you discern which ones are backed by real data.

Anyone can make a mental health app and start selling it on platforms. One study, published in the journal Anxiety and Depression in 2017, found that among 52 apps offered for anxiety, two-thirds did not have any healthcare professionals involved in their development. Less than four percent were rigorously tested. For apps that undergo clinical testing, often the research is done by researchers employed by the company selling the app.

There also aren’t robust government regulators checking these claims. The U.S. Food and Drug Administration does have jurisdiction over health apps, but the risks of using them are so low and the apps so numerous that the agency rarely intervenes unless a particularly egregious app comes along.

What they found — Goldberg’s team finds that researchers trying to gauge the effectiveness of mental-health apps are also inconsistent in how they go about it. When researchers compared the apps to other treatments, they rarely compare them to in-person therapy, for example — a strange oversight.

The meta-review found a number of other issues. For example, the research included rarely addressed how people engage with the apps being studied. In all the studies examined, the participants used the app for however long the study lasted. But when people use mental health apps in the real world, the drop-out rate is high.

“It’s a huge issue,” says Goldberg. “Even in a study, people often don’t stay with these things.”

He notes another problem with interpreting meta-analyses in general. When studies show no effect, researchers might not publish the data, leaving an “overly optimistic estimate of the effect.”

On a more positive note, mental health apps do consistently show evidence of “small magnitude effects” — some relief for many of the users — across mental health issues.

Goldberg says this is not surprising; there has to be a reason behind their popularity, after all.

“I’m sure that some claims are being overstated,” he says, “but I also think it’s likely that a lot of products that are out there are helpful for people.”

“If you take a look at [the mediation app] Headspace as an example, millions of people are using their products,” he says, “and I think for meditation apps to have millions of people using it, people have to be benefiting somewhat.”

“Meditation is not that much fun for most of us,” Goldberg adds.

How They Did It — Goldberg and his co-authors conducted a meta-review, which is sort of a meta-analysis of meta-analyses. A meta-analysis systematically merges the findings of several independent research trials to get a bigger picture of what they all studied. A meta-review does the same, but for meta-analyses.

They gathered 14 meta-analyses, each involving at least four studies. Altogether, they included 145 randomized controlled trials of mental health apps, plus some on text message-based inventions that did not use any single app. In total, the research included 47,940 participants.

Some examined research on the effectiveness of apps for relief from depression or anxiety. Others studied apps’ ability to help people quit smoking or improve their general mental wellbeing. They also examined various kinds of apps: guided meditation, texting with a person, texting with a chatbot, and do-it-yourself programs.

All the apps had one thing in common: You needed a smartphone; telehealth or in-person therapies were left out of this analysis.

What’s next — Goldberg explains that the disparate results for each app may be to do with the newness of the technology itself — they might just not have been around long enough or used enough to get a good sense of how well they work in the lab or the real world. But time will tell.

“I think there’s some fantastic meta-analytic work and there’s some fantastic [random clinical trials] happening,” he says. “I think the literature is just young.”

“If you think about that, we’ve been studying psychotherapy for a hundred years. We’ve been studying mobile phone-based interventions for maybe 20 years,” he adds. That’s some catching up to do.