Science's Reproducibility Crisis

Science Has a Reproducibility Crisis

In May of 2016, a survey by Nature revealed that over two thirds of researchers surveyed had tried and failed to reproduce the results of another scientist’s study. Over half of them had been unable to reproduce their own results. Fifty two percent of researchers polled said there was a “significant crisis” of reproducibility.

This is a big deal. The ability to replicate the results of studies is crucial to both scientific integrity and progress. Clinical researchers, for example, depend on reliable results from prior trials to form the building blocks of new drug advancements. In the field of cancer biology, merely 10% of results from published literature were found to be reproducible. Meanwhile, the credibility of scientific literature is understandably compromised by dubious, often sensational findings.

The root of the problem, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge, stems from today’s scientific culture. As quoted in BBC, she cites “a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

Others blame a pressure to publish. There has also been, in recent years, doubt cast on the integrity of the peer review process, especially with regard to climate science. Whatever the culprit, plans to combat issues of reproducibility are emerging. Nature has developed a checklist to serve as guidelines for authors submitting writing to the publication. Efforts shouldn’t end there, the journal argues. Reform at all levels of the scientific process could go a long way:

“Renewed attention to reporting and transparency is a small step. Much bigger underlying issues contribute to the problem, and are beyond the reach of journals alone. Too few biologists receive adequate training in statistics and other quantitative aspects of their subject. Mentoring of young scientists on matters of rigour and transparency is inconsistent at best. In academia, the ever increasing pressures to publish and chase funds provide little incentive to pursue studies and publish results that contradict or confirm previous papers. Those who document the validity or irreproducibility of a published piece of work seldom get a welcome from journals and funders, even as money and effort are wasted on false assumptions.
Tackling these issues is a long-term endeavour that will require the commitment of funders, institutions, researchers and publishers. It is encouraging that NIH institutes have led community discussions on this topic and are considering their own recommendations. We urge others to take note of these and of our initiatives, and do whatever they can to improve research reproducibility.”

Yes, it's a problem. All of the factors above are acting (which is the usual answer - embrace the healing power of "and"). You don't make a name repeating some else's seminal works, and granting agencies are reluctant to fund "repeat" research.

If we want more reproducible science, we need to make room for it in funding. It's really not as bad as it sounds. If reproducibility were made a requirement for publication, a lot less garbage would be funded.