If a job is worth doing, it is worth doing twice

In February, Facebook founder Mark Zuckerberg visited my university to announce the Breakthrough Prize in Life Sciences. As someone who hopes for a career in biomedical research, I applaud these awards. Biomedical science, when practised correctly, is a wellspring of knowledge, innovation and human health. But is it practised correctly? And if not, how does it need to change? In recent years, it has become clear that biomedical science is plagued by findings that cannot be reproduced. This wastes grant money and hinders development of new treatments and cures. The irreproducibility epidemic is exacerbated by a funding structure that rewards publications above all else, whether they are reproducible or not. Science as a system should place more importance on reproducibility. Not every paper needs to be medically relevant, but at the very least they should all be reproducible. Reproducibility separates science from mere anecdote. Some journals already offer to publish replication studies, and there are nascent projects aimed at reproducing work in individual fields from disease to psychology. But these are stopgap solutions. A more comprehensive answer is required. As a start, funding agencies should tie grant funding to replication. A portion of their budget could be set aside to pay for independent replication studies. If a paper cannot be replicated, the authors should be required to amend or retract it. Funding agencies would then consider a principal investigator’s history of reproducibility in grant reviews. This key reform would ensure that career success rests on the validity of findings rather than whether they are published in trendy journals. Many scientists will object to this proposal, and for many reasons. It sounds complicated. Scientists would apply for grants, conduct research and publish their findings much as they do now. The only difference is that replication studies would be more plentiful and more would be published. Each replication would be electronically linked to the original paper and boost its validity. Studies that could not be reproduced would be marked as such. Not all non-reproducible work is flawed, of course, but a red flag against a paper, accompanied by an amendment from the original authors noting discrepancies in methodology and analysis that might explain why it was not reproduced, would become a normal part of scientific discourse. Science should work through open dialogue, accessible to everyone. It would slow the dissemination of novel research. Only modestly, and the time lost would be trivial compared with the time currently lost chasing false leads. Who would do the replication studies? People are already doing them. Under my proposal, more of these studies would be published. More people will be needed, and if granting agencies provided funds, more scientists would do replication studies. Replication studies are hard. Sharing reagents and methods is not that difficult, and it is already mandated by funding agencies. Unfortunately, many do not comply. My proposal would encourage original authors to work with replicating groups and make the studies easier, because each replication would bolster the original authors’ reputation. Publishing amendments would take a lot of time. It would certainly take more time than is currently spent on it, but scientists already have similar conversations by e-mail, at conferences and in lab meetings. If publishing amendments were explicitly rewarded, people would take the time to do it. Science already self-corrects. True, but the timescale is long and the associated waste is vast. Even outright fraud can take decades to come to light, and negative results rarely come to light at all. Under my proposal, negative results would be valued and valuable. But there is no money available. This reform should save money by redirecting spending towards science that is reproducible. Society will receive a higher return on its investment in the form of treatments and cures. This proposal is too radical. The same was once said about open-access publication. Yet that is quickly becoming the norm thanks to policy changes by funding agencies. The US National Institutes of Health and other granting agencies will never adopt this policy. National governments, which fund granting agencies, could impose the reform and ensure that agencies adopt it. Acknowledging irreproducibility in science undermines public trust. Trust is already being undermined. It is best that we reverse the tide and restore that trust by practising transparency. It would discourage high-risk, high-reward science, which is less likely to be reproducible. There would be no requirement that a study be replicated before publication. If a risky paper ends up being reproducible, everybody wins. But if repeated attempts to reproduce it fail, it is not a valid finding — there is no reward to society. Why pretend otherwise? In 1953, then US president Dwight Eisenhower said that every dollar spent on war “signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and not clothed”. The same is true for every dollar spent on science that cannot be replicated. We can do better. ■