How to evaluate “promising” coronavirus treatments like a scientist

As the pandemic continues, scientists are racing against time to find a treatment for Covid-19.

The good news here is that there are a number of trials going on around the world to make such a medicine materialize.

The bad news is that there are so many going on all at one time, that it is hard to tell to which projects are worth putting your hope into, and which aren't strong enough to deserve your time.

But armed with these four, science-backed strategies, you can start to make sense of which trials to pay attention to.

The United States' Food and Drug Administration (FDA) has not recognized any drug as a treatment or cure for Covid-19.

But the the speed of scientific response to the call for a treatment is "very encouraging," says Nathan Peiffer-Smadja, a physician and infectious disease researcher at Imperial College London.

Peiffer-Smadja is also one of the authors on a medRxiv pre-print paper that evaluated 115 clinical trials for Covid-19 treatments. This paper has not been peer-reviewed. But it does provide a snapshot of some of the ongoing trials, and analyzes how well they are designed.

"[It's] very difficult to design a trial in these challenging times, but many teams met this challenge," he tells Inverse.

Part of the problem is that science is struggling to keep the pace with the public conversation around the novel coronavirus. Some drugs are gaining popularity as "coronavirus treatments" long before they have actually shown any efficacy, says Andre Kalil, a professor at the University of Nebraska's Department of Internal Medicine.

"This is a hard moment," Kalil tells Inverse.

"There are all kinds of promising news about drugs that haven’t even been tested in real trials. That’s really a problem because people can be misled by pseudoscience."

We spoke to three scientists who are all working in the field right now to come up with the four questions you should ask when evaluating the merits of any scientific study on coronavirus treatments.

4. Where was the study published?

3. Is the trial randomized and controlled?

2. What size is the study?

1. Why is the drug being tested?

Here is why applying these four strategies can help you sort through the noise:

4. Where was the study published?

Whenever you read a headline about a treatment or cure for Covid-19, make sure the cited study is peer-reviewed and published in a reputable scientific journal.

This may seem like a given, but it is not always the case, Kalil says.

"Most of these studies that are going viral very fast on social media are papers published in obscure journals that really don't do a peer-review process," Kalil says.

"I think this is dangerous, and it's already hurting people because it brings false promises and false hope."

Take the example of chloroquine, a drug currently undergoing clinical trials as a potential Covid-19 treatment.

Although a peer-reviewed study published in The International Journal of Antimicrobial Agents touted the antimalarial's promise, it initially gained widespread public notoriety via Elon Musk's Twitter feed and a Google document. That document was far from a scientific study, but it was discussed on social media and on the news as if it were.

In other cases, scientists' papers might be published on pre-print servers like bioRxiv or medRxiv (as Peiffer-Smadja's did). These pre-print servers get novel science out fast, and encourage members of the scientific community's comment. But they are not subject to the same rigorous peer-review process as a journal article.

The servers try to make that reality clear: medRxiv warns users that any papers published on Covid-19 treatments have not been peer reviewed, so they cannot be taken as a guide to treatment.

If the study was published in a journal and peer-reviewed, it still helps to do a bit of extra homework on the journal, Kalil says. Some journals are considered more rigorous, and thus more likely to publish solid science, than others.

Check how long the journal has been around, and see what else they have published in regards to Covid-19. You can also look at the journal's impact factor, a measure of how often articles published there are cited by other scientific papers.

Check to see if the study has been mentioned by a leading health organization like the World Health Organization, or the US Centers for Disease Control and Prevention, too, he says.

"If there's anything that's going to impact the treatment of Covid-19, you're going to see it on the CDC site or the WHO site. They're digging 24/7."

3. Look for the two magic words

Whenever you're looking at a news report or a clinical trial itself, you should be on the lookout for two words: randomized and controlled.

"This is the minimum you need to know if something is reliable," Kalil says.

"Look for those two words and at least you will know that the people behind the study were trying to do the right science."

Randomized means that the treatment (in this case, a trial of a drug for Covid-19) is assigned to someone at random. The scientists don't pick who gets the treatment and who doesn't.

Controlled means that you are comparing a treatment group to another group of people who are receiving either the existing standard of care, no treatment at all, or in some cases, a placebo. They are the control group.

Another phrase you can look out for is "double blind." That means that neither the researchers, nor the subjects, know who is in the treatment group and who is in the control group. This extra step helps reduce bias when it comes to analyzing whether the treatment worked or not, and in whom.

If you see those words, "it brings more robustness to the study," Kalil says.

You may also see other phrases, like "open-label," which means that both the trial subjects and the scientists know who is getting what.

Sometimes, the nature of the medication means it is difficult to design a double-blind trial, Kalil says. The double-blind design adds validity to the results, but the most important factors in the trial of any treatment are randomization and control.

"The word 'promising' is very important in the context of real science. That’s what’s going to save people."

2. What size is the study?

The actual size of the study also matters, Angela Rassmussen, a virologist at Columbia University, tells Inverse.

"The key things I look for in a clinical trial are the controls and the group sizes," she says.

If a trial involves a large group of people, there is a higher chance that the results are not down to individual anomaly.

She suggests you ask yourself this question:

"Was the trial large enough to account for individual variation and assess that the results would be true in a diverse population of people, and not just for a few outliers?"

1. Ask yourself, why is this drug being tested?

Take a quick look at the rationale the scientists give for testing certain therapies, Peiffer-Smadja says.

"We have seen in the review trials testing stem cell therapies, with very limited scientific background, homeopathy, or dietary supplements," he says.

"We do not have any data suggesting that these could be effective in a viral disease that is quite severe."

Typically, when a treatment enters the clinical-trial stage, it is because it has shown some preliminary promise in animal studies or in cell culture.

This was the case for chloroquine, and another drug under investigation for treating Covid-19, Remdesivir (interestingly, it was first investigated as a treatment for Ebola).

If there is solid preclinical evidence that the medication holds promise for humans, then that is encouraging. It is not enough to prove these treatments are effective, but it does give a sense of why any drug is on the table.

A "great hope"

In the case of the novel coronavirus, there is a lot to be hopeful about when it comes to the search for a drug treatment. The scientific community has rallied to the call in a huge way — and they are working together.

For example, the WHO has launched a project called SOLIDARITY trial, a "mega trial" including clinical trials conducted in at least ten different countries.

Peiffer-Smadja is involved in the European arm of the SOLIDARITY trial. He calls it a "great hope."

Any clinical trials that adhere to high standards should give us hope, Kalil says. They are intended to turn "promise" into real treatments, after all.

But it is early days. Knowing a little about the way clinical trials work now may help you recognize a real contender for tackling Covid-19 when you see it.

"My hope is that in the next few weeks we should have results coming out and we will understand what works and what doesn’t," Kalil says.

"The word 'promising' is very important in the context of real science. That’s what’s going to save people."