New study raises serious questions about reliability of the science used to develop cancer treatments.
(Photo from Science News, December 7, 2012)
After eight years, a project that tried to reproduce the results of key cancer biology studies has finally concluded. And its findings suggest that like research in the social sciences, cancer research has a replication problem.
Researchers with the Reproducibility Project: Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012. But only a quarter of those experiments were able to be reproduced, the team reports in two papers published December 7 in eLife.
The researchers couldn’t complete the majority of experiments because the team couldn’t gather enough information from the original papers or their authors about methods used, or obtain the necessary materials needed to attempt replication.
What’s more, of the 50 experiments from 23 papers that were reproduced, effect sizes were, on average, 85 percent lower than those reported in the original experiments. Effect sizes indicate how big the effect found in a study is. For example, two studies might find that a certain chemical kills cancer cells, but the chemical kills 30 percent of cells in one experiment and 80 percent of cells in a different experiment. The first experiment has less than half the effect size seen in the second one.
The team also measured if a replication was successful using five criteria. Four focused on effect sizes, and the fifth looked at whether both the original and replicated experiments had similarly positive or negative results, and if both sets of results were statistically significant. The researchers were able to apply those criteria to 112 tested effects from the experiments they could reproduce. Ultimately, just 46 percent, or 51, met more criteria than they failed, the researchers report.
“The report tells us a lot about the culture and realities of the way cancer biology works, and it’s not a flattering picture at all,” says Jonathan Kimmelman, a bioethicist at McGill University in Montreal. He coauthored a commentary on the project exploring the ethical aspects of the findings.
“It’s worrisome if experiments that cannot be reproduced are used to launch clinical trials or drug development efforts,” Kimmelman says. If it turns out that the science on which a drug is based is not reliable, “. . . it means that patients are needlessly exposed to drugs that are unsafe and that really don’t even have a shot at making an impact on cancer.”
“This points to the importance of primary prevention efforts,” says Greg Anderson, President of Cancer Recovery Group. “Patients need to understand that this study raises serious questions about reliability of the science used to develop cancer treatments.
“High-level nutrition, daily exercise, personal stress management—all the lifestyle evidence-based preventative measures—are the key,” says Anderson, himself a lung cancer survivor. “Cancer is largely a preventable illness. We need to act on primary prevention protocols as the basis of ending cancer in our lifetimes.”