Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions

Scientific code is different from production software. Scientific code, by producing results that are then analyzed and interpreted, participates in the elaboration of scientific conclusions. This imposes specific constraints on the code that are often overlooked in practice. We articulate, with a small example, five characteristics that a scientific code in computational science should possess: re-runnable, repeatable, reproducible, reusable, and replicable. The code should be executable (re-runnable) and produce the same result more than once (repeatable); it should allow an investigator to reobtain the published results (reproducible) while being easy to use, understand and modify (reusable), and it should act as an available reference for any ambiguity in the algorithmic descriptions of the article (replicable).

[1]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[2]  Christian S. Collberg,et al.  Repeatability in computer systems research , 2016, Commun. ACM.

[3]  Lorena A. Barba,et al.  Reproducible and Replicable Computational Fluid Dynamics: It’s Harder Than You Think , 2017, Computing in Science & Engineering.

[4]  Ludovic Courtès,et al.  Reproducible and User-Controlled Software Environments in HPC with Guix , 2015, Euro-Par Workshops.

[5]  Daniel S. Katz,et al.  Journal of Open Source Software (JOSS): design and first-year review , 2017, PeerJ Comput. Sci..

[6]  Kai Diethelm,et al.  The Limits of Reproducibility in Numerical Simulation , 2012, Computing in Science & Engineering.

[7]  David Defour,et al.  Numerical reproducibility for the parallel reduction on multi- and many-core architectures , 2015, Parallel Comput..

[8]  John P. A. Ioannidis,et al.  Reproducible Research Practices and Transparency across the Biomedical Literature , 2016, PLoS biology.

[9]  Matthias Schwab,et al.  Making scientific computations reproducible , 2000, Comput. Sci. Eng..

[10]  Lorena A. Barba,et al.  Reproducible and replicable CFD: it's harder than you think , 2016, 1605.04339.

[11]  Andrew P. Davison,et al.  Learning from the Past: Approaches for Reproducibility in Computational Neuroscience , 2013 .

[12]  Hans Knutsson,et al.  Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates , 2016, Proceedings of the National Academy of Sciences.

[13]  N. T. Carnevale,et al.  Translating network models to parallel hardware in NEURON , 2008, Journal of Neuroscience Methods.

[14]  F. Dominici,et al.  Reproducible epidemiologic research. , 2006, American journal of epidemiology.

[15]  Arian Maleki,et al.  Reproducible Research in Computational Harmonic Analysis , 2009, Computing in Science & Engineering.

[16]  Jill P Mesirov,et al.  Accessible Reproducible Research , 2010, Science.

[17]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[18]  Victoria Stodden,et al.  Reproducible Research , 2019, The New Statistics with R.

[19]  Vern Paxson,et al.  The Matter of Heartbleed , 2014, Internet Measurement Conference.

[20]  Michelle Schwalbe,et al.  Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop , 2016 .

[21]  Cyrille Rossant,et al.  Sustainable computational science: the ReScience initiative , 2017, PeerJ Comput. Sci..

[22]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .