A statistical definition for reproducibility and replicability

Everyone agrees that reproducibility and replicability are fundamental characteristics of scientific studies. These topics are attracting increasing attention, scrutiny, and debate both in the popular press and the scientific literature. But there are no formal statistical definitions for these concepts, which leads to confusion since the same words are used for different concepts by different people in different fields. We provide formal and informal definitions of scientific studies, reproducibility, and replicability that can be used to clarify discussions around these concepts in the scientific and popular press.

[1]  Michael C. Frank,et al.  Response to Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[2]  Yang Li,et al.  G-protein-coupled receptor GPR21 knockout mice display improved glucose tolerance and increased insulin response. , 2012, Biochemical and biophysical research communications.

[3]  Donald A. Wilson,et al.  ApoE-Directed Therapeutics Rapidly Clear β-Amyloid and Reverse Deficits in AD Mouse Models , 2012, Science.

[4]  Lucas N Joppa,et al.  Government: Plan for ecosystem services. , 2016, Science.

[5]  G. Nahler,et al.  Replication Study , 2017 .

[6]  C. Ball,et al.  Repeatability of published microarray gene expression analyses , 2009, Nature Genetics.

[7]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[8]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[9]  C. Glenn Begley,et al.  Raise standards for preclinical cancer research , 2012 .

[10]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[11]  K. Coombes,et al.  Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology , 2009, 1010.1092.

[12]  B. Birmaher,et al.  Psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): a replication study. , 1999, Journal of the American Academy of Child and Adolescent Psychiatry.

[13]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[14]  M. McCarthy,et al.  Meta-analysis of genome-wide association data and large-scale replication identifies additional susceptibility loci for type 2 diabetes , 2008, Nature Genetics.

[15]  Thomas C. Herndon,et al.  Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff , 2014 .

[16]  Miriam Scadeng,et al.  G protein-coupled receptor 21 deletion improves insulin sensitivity in diet-induced obese mice. , 2012, The Journal of clinical investigation.

[17]  Stephan Lewandowsky,et al.  Learning from mistakes in climate research , 2016, Theoretical and Applied Climatology.

[18]  Jeffrey T. Leek,et al.  Is most published research really false? , 2016, bioRxiv.

[19]  Min Jae Lee,et al.  Enhancement of Proteasome Activity by a Small-Molecule Inhibitor of Usp14 , 2010, Nature.

[20]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[21]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[22]  Timothy D. Wilson,et al.  Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[23]  Jeffrey T. Leek,et al.  Opinion: Reproducible research can still be wrong: Adopting a prevention approach , 2015, Proceedings of the National Academy of Sciences.