Sustainable computational science : the

Computer science o‚ers a large set of tools for prototyping, writing, running, testing, validating, sharing and reproducing results, however computational science lags behind. In the best case, authors may provide their source code as a compressed archive and they may feel con€dent their research is reproducible. But this is not exactly true. Jonathan Buckheit and David Donoho proposed more than two decades ago that an article about computational results is advertising, not scholarship. ‘e actual scholarship is the full so‰ware environment, code, and data that produced the result. ‘is implies new workƒows, in particular in peer-reviews. Existing journals have been slow to adapt: source codes are rarely requested, hardly ever actually executed to check that they produce the results advertised in the article. ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research can be replicated from its description. To achieve this goal, the whole publishing chain is radically di‚erent from other traditional scienti€c journals. ReScience resides on GitHub where each new implementation of a computational study is made available together with comments, explanations, and so‰ware tests.

[1]  Ian M. Mitchell,et al.  Best Practices for Scientific Computing , 2012, PLoS biology.

[2]  Joseph Stachelek,et al.  [Re] Least-cost modelling on irregular landscape graphs , 2016 .

[3]  Yaroslav O. Halchenko,et al.  Four aspects to make science open “by design” and not as an after-thought , 2015, GigaScience.

[4]  Tiina Manninen,et al.  Reproducibility and Comparability of Computational Models for Astrocyte Calcium Excitability , 2017, Front. Neuroinform..

[5]  Nicolas P. Rougier,et al.  A long journey into reproducible computational neuroscience , 2015, Front. Comput. Neurosci..

[6]  Daniel S. Katz,et al.  Journal of Open Source Software (JOSS): design and first-year review , 2017, PeerJ Comput. Sci..

[7]  Philip J. Guo,et al.  CDE: Using System Call Interposition to Automatically Create Portable Software Packages , 2011, USENIX Annual Technical Conference.

[8]  Christian Collberg,et al.  Measuring Reproducibility in Computer Systems Research , 2014 .

[9]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[10]  Nicolas P. Rougier,et al.  [Re] Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study , 2015 .

[11]  Ludovic Courtès,et al.  Reproducible and User-Controlled Software Environments in HPC with Guix , 2015, Euro-Par Workshops.

[12]  Christian S. Collberg,et al.  Repeatability in computer systems research , 2016, Commun. ACM.

[13]  Frank Yellin,et al.  The Java Virtual Machine Specification , 1996 .

[14]  J. Bower 20 Years of Computational Neuroscience , 2013, Springer Series in Computational Neuroscience.

[15]  MalekiArian,et al.  Reproducible Research in Computational Harmonic Analysis , 2009 .

[16]  John P. A. Ioannidis,et al.  Reproducible Research Practices and Transparency across the Biomedical Literature , 2016, PLoS biology.

[17]  J. Ioannidis Why Most Published Research Findings Are False , 2019, CHANCE.

[18]  Mehdi Khamassi,et al.  [Re] Speed/accuracy trade-off between the habitual and the goal-directed process , 2016 .

[19]  Arian Maleki,et al.  Reproducible Research in Computational Harmonic Analysis , 2009, Computing in Science & Engineering.

[20]  Juliana Freire,et al.  noWorkflow: Capturing and Analyzing Provenance of Scripts , 2014, IPAW.

[21]  Konrad Hinsen Writing Software Specifications , 2015, Computing in Science & Engineering.

[22]  Krishna T. Malladi,et al.  CoNDA , 2019, Proceedings of the 46th International Symposium on Computer Architecture.

[23]  Andrew P. Davison Automated Capture of Experiment Context for Easier Reproducibility in Computational Research , 2012, Computing in Science & Engineering.

[24]  Alex M. Warren Repeatability and Benefaction in Computer Systems Research — A Study and a Modest Proposal , 2015 .

[25]  Nicole Janz Online Appendix for: Bringing the Gold Standard Into the Class Room: Replication in University Teaching , 2017 .

[26]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[27]  Greg Wilson,et al.  Software Carpentry: lessons learned , 2013, F1000Research.

[28]  François-Xavier Coudert,et al.  Reproducible Research in Computational Chemistry of Materials , 2017 .

[29]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[30]  M. Baker 1,500 scientists lift the lid on reproducibility , 2016, Nature.

[31]  Lorena A. Barba,et al.  Reproducible and replicable CFD: it's harder than you think , 2016 .

[32]  Charles Anderson,et al.  Docker , 2015, IEEE Softw..

[33]  Jeffrey Perkel,et al.  Democratic databases: science on GitHub , 2016, Nature.