Reproducibility in Science: Improving the Standard for Basic and Preclinical Research

Medical and scientific advances are predicated on new knowledge that is robust and reliable and that serves as a solid foundation on which further advances can be built. In biomedical research, we are in the midst of a revolution with the generation of new data and scientific publications at a previously unprecedented rate. However, unfortunately, there is compelling evidence that the majority of these discoveries will not stand the test of time. To a large extent, this reproducibility crisis in basic and preclinical research may be as a result of failure to adhere to good scientific practice and the desperation to publish or perish. This is a multifaceted, multistakeholder problem. No single party is solely responsible, and no single solution will suffice. Here we review the reproducibility problems in basic and preclinical biomedical research, highlight some of the complexities, and discuss potential solutions that may help improve research quality and reproducibility.

[1]  John P. A. Ioannidis,et al.  Assessing value in biomedical research: the PQRST of appraisal and reward. , 2014, JAMA.

[2]  Henry I. Miller,et al.  Are Medical Articles True on Health, Disease? Sadly, Not as Often as You Might Think , 2014 .

[3]  S. Perrin.  Preclinical research: Make mouse studies work , 2014, Nature.

[4]  W. Kraus.  Editorial: Do you see what I see? Quality, reliability, and reproducibility in biomedical research. , 2014, Molecular endocrinology.

[5]  E. Diamandis,et al.  False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. , 2014, Clinical Chemistry.

[6]  Birte U. Forstmann,et al.  Rewarding high-power replication research , 2014, Cortex.

[7]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[8]  John P A Ioannidis,et al.  Improving the drug development process: more not less randomized trials. , 2014, JAMA.

[9]  Harlan M Krumholz,et al.  Increasing value and reducing waste: addressing inaccessible research , 2014, The Lancet.

[10]  D. Moher,et al.  Reducing waste from incomplete or unusable reports of biomedical research , 2014, The Lancet.

[11]  Jonathan M. Kagan,et al.  Increasing value and reducing waste in biomedical research regulation and management , 2014, The Lancet.

[12]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[13]  Iain Chalmers,et al.  How to increase value and reduce waste when research priorities are set , 2014, The Lancet.

[14]  Ubbo Visser,et al.  Facilitating transparency in spinal cord injury studies using data standards and ontologies , 2014, Neural regeneration research.

[15]  D. Snow.  Commentary on: “Facilitating transparency in spinal cord injury studies using data standards and ontologies” , 2014, Neural regeneration research.

[16]  Katie Lidster,et al.  Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies , 2014, PLoS biology.

[17]  S. Goodman,et al.  Raw data from clinical trials: within reach? , 2013, Trends in pharmacological sciences.

[18]  Kevin W Boyack,et al.  A list of highly influential biomedical researchers, 1996–2011 , 2013, European journal of clinical investigation.

[19]  Benjamin Haibe-Kains,et al.  Inconsistency in large pharmacogenomic studies , 2013, Nature.

[20]  E. Hayden.  Personalized cancer treatments suffer setback , 2013 .

[21]  Mina Bissell,et al.  Reproducibility: The risks of the replication drive , 2013, Nature.

[22]  William A. Thompson,et al.  Assessing the validity and reproducibility of genome-scale predictions , 2013, Bioinform..

[23]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[24]  C. Begley.  The test of time and editorial responsibility , 2013, Pigment Cell & Melanoma Research.

[25]  Sven Kepes,et al.  How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology? , 2013, Industrial and Organizational Psychology.

[26]  Meredith Wadman,et al.  NIH mulls rules for validating key results , 2013, Nature.

[27]  M. Ritskes-Hoitinga,et al.  Progress in Using Systematic Reviews of Animal Studies to Improve Translational Research , 2013, PLoS medicine.

[28]  J. Grimshaw,et al.  Threats to Validity in the Design and Conduct of Preclinical Efficacy Studies: A Systematic Review of Guidelines for In Vivo Animal Experiments , 2013, PLoS medicine.

[29]  J. Ioannidis,et al.  Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases , 2013, PLoS biology.

[30]  V. Stodden,et al.  Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals , 2013, PloS one.

[31]  Daniel A Erlanson,et al.  Learning from our mistakes: the 'unknown knowns' in fragment screening. , 2013, Bioorganic & medicinal chemistry letters.

[32]  Suzanne K. Linder,et al.  A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic , 2013, PloS one.

[33]  Richard Van Noorden.  Meeting targets lab lapses , 2013, Nature.

[34]  Begley Cg,et al.  Ocean science: Arctic sea ice needs better forecasts , 2013, Nature.

[35]  Anne L Plant,et al.  Translating stem cell research from the bench to the clinic: a need for better quality data. , 2013, Stem cells and development.

[36]  P. Houghton,et al.  A Proposal Regarding Reporting of In Vitro Testing Results , 2013, Clinical Cancer Research.

[37]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[38]  L. Standing,et al.  Is Intelligence Enhanced by Letter Priming? A Failure to Replicate the Results of Ciani and Sheldon (2010) , 2013, Psychological reports.

[39]  K. Baggerly.  More data, please! , 2013, Clinical chemistry.

[40]  Kenneth W Witwer,et al.  Data submission and quality in microarray-based microRNA profiling. , 2013, Clinical chemistry.

[41]  David L. Vaux,et al.  Research methods: Know when your numbers are significant , 2012, Nature.

[42]  J. Ioannidis,et al.  Research grants: Conform and be funded , 2012, Nature.

[43]  George Poste,et al.  Biospecimens, biomarkers, and burgeoning data: the imperative for more rigorous research standards. , 2012, Trends in molecular medicine.

[44]  P. Hall,et al.  Quality really matters: the need to improve specimen quality in biomedical research , 2012, Histopathology.

[45]  S. Lazic,et al.  Improving basic and translational science by accounting for litter-to-litter variation in animal models , 2012, BMC Neuroscience.

[46]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[47]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[48]  S. Lazic,et al.  A call for transparent reporting to optimize the predictive value of preclinical research , 2012, Nature.

[49]  Chris Harbron,et al.  In search of preclinical robustness , 2012, Nature Reviews Drug Discovery.

[50]  James Taylor,et al.  Next-generation sequencing data interpretation: enhancing reproducibility and accessibility , 2012, Nature Reviews Genetics.

[51]  Ian M. Mitchell,et al.  Reproducible research for scientific computing: Tools and strategies for changing the culture , 2012, Computing in Science & Engineering.

[52]  Ray Gibson,et al.  Editorial: Reproducible Research in Medical Imaging , 2012, Molecular Imaging and Biology.

[53]  Jeffrey R. Spies,et al.  Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability , 2012, 1205.4251.

[54]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[55]  John P A Ioannidis,et al.  Reversals of established medical practices: evidence to abandon ship. , 2012, JAMA.

[56]  A. Casadevall,et al.  Reforming Science: Methodological and Cultural Reforms , 2011, Infection and Immunity.

[57]  A. Casadevall,et al.  Reforming Science: Structural Reforms , 2011, Infection and Immunity.

[58]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[59]  J. Ioannidis,et al.  Improving Validation Practices in “Omics” Research , 2011, Science.

[60]  John P. A. Ioannidis,et al.  More time for research: Fund people not projects , 2011, Nature.

[61]  J. Ioannidis,et al.  Public Availability of Published Research Data in High-Impact Journals , 2011, PloS one.

[62]  S. Stanley Young,et al.  Deming, data and observational studies , 2011 .

[63]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[64]  D. Fanelli.  “Positive” Results Increase Down the Hierarchy of the Sciences , 2010, PloS one.

[65]  D. Howells,et al.  Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy , 2010, PLoS biology.

[66]  K. Coombes,et al.  Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology , 2009, 1010.1092.

[67]  P. Glasziou,et al.  Avoidable waste in the production and reporting of research evidence , 2009, The Lancet.

[68]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[69]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[70]  Heejung Bang,et al.  Cereal-induced gender selection? Most likely a multiple testing false positive , 2009, Proceedings of the Royal Society B: Biological Sciences.

[71]  A. Field,et al.  Test of Time , 2009, Clinical child psychology and psychiatry.

[72]  C. Ball,et al.  Repeatability of published microarray gene expression analyses , 2009, Nature Genetics.

[73]  Chad J. Creighton,et al.  MDA-MB-435 cells are derived from M14 Melanoma cells––a loss for breast cancer, but a boon for melanoma research , 2007, Breast Cancer Research and Treatment.

[74]  P. Sandercock,et al.  Comparison of treatment effects between animal experiments and clinical trials: systematic review , 2006, BMJ : British Medical Journal.

[75]  D. Redelmeier,et al.  Translation of research evidence from animals to humans. , 2006, JAMA.

[76]  F. Dominici,et al.  Reproducible epidemiologic research. , 2006, American journal of epidemiology.

[77]  C. Begley,et al.  Anti-Epo receptor antibodies do not predict Epo receptor expression. , 2006, Blood.

[78]  John P A Ioannidis,et al.  Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results , 2006, Cancer Epidemiology Biomarkers & Prevention.

[79]  David W Howells,et al.  Pooling of Animal Experimental Data Reveals Influence of Study Design and Publication Bias , 2004, Stroke.

[80]  Christian A. Rees,et al.  Systematic variation in gene expression patterns in human cancer cell lines , 2000, Nature Genetics.

[81]  R. Nickerson.  Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[82]  N. Dawson.  Ensuring scientific integrity , 1987, Nature.

[83]  Clifford R. Mynatt,et al.  Confirmation Bias in a Simulated Research Environment: An Experimental Study of Scientific Inference , 1977 .

[84]  R. Rosenthal,et al.  A LONGITUDINAL STUDY OF THE EFFECTS OF EXPERIMENTER BIAS ON THE OPERANT LEARNING OF LABORATORY RATS. , 1964, Journal of psychiatric research.

[85]  E H McClelland,et al.  MORE DATA. , 1927, Science.

[86]  Raymond J Winquist,et al.  The fall and rise of pharmacology--(re-)defining the discipline? , 2014, Biochemical pharmacology.

[87]  F. Collins,et al.  NIH plans to enhance reproducibility , 2014 .

[88]  Michael Williams,et al.  Replicated, replicable and relevant-target engagement and pharmacological experimentation in the 21st century. , 2014, Biochemical pharmacology.

[89]  B. Ruggeri,et al.  Animal models of human disease: challenges in enabling translation. , 2014, Biochemical pharmacology.

[90]  Michael J Marino,et al.  The use and misuse of statistical methodologies in pharmacology research. , 2014, Biochemical pharmacology.

[91]  Alan F. Karr,et al.  Deming, data and observational studies: A process out of control and needing fixing , 2013 .

[92]  R. Schekman.  How journals like Nature, Cell and Science are damaging science , 2013 .

[93]  Maryann E Martone,et al.  A survey of the neuroscience resource landscape: perspectives from the neuroscience information framework. , 2012, International review of neurobiology.

[94]  Rob J. Hyndman,et al.  Encouraging replication and reproducible research , 2010 .

[95]  Thomas E Novotny,et al.  US Department of Health and Human Services: a need for global health leadership in preparedness and health diplomacy. , 2006, American journal of public health.

[96]  Michael R. Seringhaus.  Operators and Promoters: The Story of Molecular Biology and Its Creators , 2002, The Yale Journal of Biology and Medicine.

[97]  Jonathan Y. Richmond,et al.  Biosafety in microbiological and biomedical laboratories , 1999 .

[98]  John P A Ioannidis.  Essay Modeling the Framework for False Positive Findings Why Most Published Research Findings Are False , 2022 .