A new rating system could help you decipher conflicting scientific research

Anyone who reads health news has likely experienced information whiplash at some point. On Monday, we read stories touting a study finding small amounts of alcohol are good for you; on Tuesday, we read others suggesting no amount of alcohol is healthy. Science isn’t linear; studies of the same topic don’t always confirm each other. If you’re a reader who tries to stay up to date on the latest research in hopes of living your healthiest life, it can be confusing.

Contradicting research can also have grave public health consequences. Policymakers need good data to shape public health recommendations, and doctors need the same to properly advise their patients. Now, a team of researchers has created a system that could help cut through the noise and help both health experts and lay readers alike tell good and bad evidence apart.

What’s new — Researchers at the University of Washington’s Institute for Health Metrics and Evaluation (IHME) created the Burden of Proof Risk Function. The BPRF is a star-rating system unveiled today in the journal Nature Medicine. The system aims to provide explicit, easily understandable evidence ratings between risk-outcomes. It’s designed to “help the public decision-makers and scientific community navigate the inconsistent evidence that comes out, study by study,” Christopher Murray, Director of the IMHE, said during a press conference on the study.

How it works — First, researchers take a systematic review of the evidence for a given risk-outcome, for example, a link between vegetable consumption and colon cancer. BPRF is a meta-analysis that accounts for disagreement among study results and bias and comes up with the most conservative interpretation of the evidence. That result is converted into a star-rating system: one star for very weak evidence and five for very strong evidence.

If an analysis of the evidence linking vegetable consumption and a reduced likelihood of developing colon cancer earned four or five stars, it suggests there is solid evidence to support that link. If the same analysis awarded only one or two stars to the evidence for the link, then you could conclude that there is weak evidence for an association between vegetable consumption and a reduced likelihood of developing colon cancer.

To illustrate how the new rating system works, IMHE researchers analyzed the evidence for the links between four risk factors and specific health outcomes:

• Smoking and lung cancer
• Systolic blood pressure and ischemic heart disease
• Vegetable consumption and ischemic heart disease
• Consumption of unprocessed red meat and ischemic heart disease

The researchers amassed all the evidence they could find on each topic and then applied their BPRF meta-analysis to the pile, determining how convincing was the evidence for any of the four links.

What they found — For smoking and lung cancer, researchers evaluated “371 observations from 25 prospective cohort studies and 53 case-control studies... reported from 1980 onwards.”

They concluded the evidence for smoking as a risk factor for developing lung cancer was extremely strong, earning their highest evidence rating: five stars.

“Even taking the most conservative interpretation of the evidence — the 5th quantile risk function including between-study heterogeneity, or the BPRF — smoking dramatically increases the risk of lung cancer,” the researchers write.

“The BPRF suggests that smoking in the range of the 15th–85th percentiles of exposure raises the risk of lung cancer by an average of 106.7 percent, for an ROS of 0.73… These findings led us to classify smoking and lung cancer as a five-star risk–outcome pair.”

The evidence for a relationship between systolic blood pressure and ischemic heart disease was also very strong, earning five stars.

Digging into the details — Christian Razo, one of the researchers on the study, explained how scientists applied the BPRF to these studies. At the press conference, Razo explained: “Systolic blood pressure and ischemic heart disease is a very good example that allowed us to validate the burden of proof methodology because we knew in advance that there was a substantial body of evidence of a causal relationship [between blood pressure and heart disease],” she said. “Our results show here a very clear dose-response relationship between systolic blood pressure and ischemic heart disease supporting the widely spread consensus around this relationship, and also show that even using a very conservative measure of the risk, increasing systolic blood pressure levels increases the likelihood of developing or dying from ischemic heart disease by around 100 percent.”

Not all the risk factor-health outcome relationships were so strong. The researchers also analyzed where the evidence was surprisingly weak—for example, the relationship between vegetable consumption and ischemic heart disease.

“The relationship is not log-linear. We found that, on average, vegetable consumption was protective, with the relative risk of ischemic heart disease being 0.81 (0.74–0.89) at 100 grams per day vegetable consumption compared to 0 grams per day,” the authors wrote. “Incrementally higher levels of exposure are associated with less steep declines in relative risk compared to those at lower levels of exposure.”

In other words, while vegetable consumption confers some protection against ischemic heart disease – there’s a limit, and that limit is low.

“The BPRF suggests that vegetable consumption in the range of the 15th to the 85th percentile lowers risk of ischemic heart disease by 12.10 percent on average (ROS of 0.13).” The evidence for the relationship between vegetable consumption and ischemic heart disease relationship earned just two stars on the BPRF.

The evidence for eating unprocessed red meat and developing ischemic heart disease was even weaker. The researchers found that when using the BPRF, the relationship between the two was “essentially on the null threshold.” They write: “For unprocessed red meat and ischemic heart disease, the exposure-averaged BPRF is 0.01, essentially on the null threshold, equating to an ROS of 0.01, with a corresponding increase in risk of 1.04 percent.”

This earned the relationship two stars, though in the press conference, the researchers noted that this was the low end of two stars, whereas the evidence for a relationship between vegetable consumption and ischemic heart disease was on the higher end of two stars.

The researchers note that the ratings for links that haven’t been extensively studied are likely to be low initially and to change; topics with lots of scientific literature supporting them are less likely to change. A high rating indicates not just that the evidence is good but also that there’s a lot of good evidence.

What it means for the future — The authors stress that the BPRF is just one tool to help researchers synthesize conflicting health data into a clear, accurate takeaway.

“No quantitative assessment of the evidence can or should substitute completely for expert deliberation, but a quantitative meta-analytic approach...could be a useful input to international and national expert committee considerations,” the researchers write in their paper.

Conflicting health information can have serious consequences for public health, as well as how much trust the public has in public health officials. Hopefully, implementing the BPRF system can help change that. For example, if the system was applied to those alcohol health studies, we might be able to parse the results with more clarity and confidence.