Reproducibility in Scientific Computing

Reproducibility is widely considered to be an essential requirement of the scientific process. However, a number of serious concerns have been raised recently, questioning whether today’s computational work is adequately reproducible. In principle, it should be possible to specify a computation to sufficient detail that anyone should be able to reproduce it exactly. But in practice, there are fundamental, technical, and social barriers to doing so. The many objectives and meanings of reproducibility are discussed within the context of scientific computing. Technical barriers to reproducibility are described, extant approaches surveyed, and open areas of research are identified.

[1]  V. Stodden.  Trust your science? Open your data and code , 2011 .

[2]  Yong Zhao,et al.  Chimera: a virtual data system for representing, querying, and automating data derivation , 2002, Proceedings 14th International Conference on Scientific and Statistical Database Management.

[3]  C HutchinsonNorman,et al.  Deciding when to forget in the Elephant file system , 1999 .

[4]  Ian J. Taylor,et al.  Workflows and e-Science: An overview of workflow system features and capabilities , 2009, Future Gener. Comput. Syst..

[5]  John R. Rice,et al.  Future Research Directions in Problem Solving Environments for Computational Science , 1991, Programming Environments for High-Level Scientific Problem Solving.

[6]  Inna Kouper,et al.  Towards Sustainable Curation and Preservation: The SEAD Project's Data Services Approach , 2015, 2015 IEEE 11th International Conference on e-Science.

[7]  Dennis Gannon,et al.  Workflows for e-Science, Scientific Workflows for Grids , 2014 .

[8]  Amin Vahdat,et al.  Transparent Result Caching , 1997, USENIX Annual Technical Conference.

[9]  Renato J. O. Figueiredo,et al.  VMPlants: Providing and Managing Virtual Machine Execution Environments for Grid Computing , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[10]  LudäscherBertram,et al.  Scientific workflow management and the Kepler system , 2006 .

[11]  Rajkumar Buyya,et al.  A Taxonomy of Workflow Management Systems for Grid Computing , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[12]  Ian T. Foster,et al.  Using Provenance for Repeatability , 2013, TaPP.

[13]  Eelco Visser,et al.  Software deployment in a dynamic cloud: From device to service orientation in a hospital environment , 2009, 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing.

[14]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[15]  Matthew R. Pocock,et al.  Taverna: a tool for the composition and enactment of bioinformatics workflows , 2004, Bioinform..

[16]  David L. Donoho,et al.  A Universal Identifier for Computational Results , 2011, ICCS.

[17]  Norman C. Hutchinson,et al.  Deciding when to forget in the elephant file system , 2000, OPSR.

[18]  Lorena A Barba,et al.  The hard road to reproducibility. , 2016, Science.

[19]  Nathalie Furmento,et al.  ICENI Dataflow and Workflow: Composition and Scheduling in Space and Time , 2003 .

[20]  Douglas Thain,et al.  Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids , 2012, SWEET '12.

[21]  Theo Lynn,et al.  Quality Assurance for Open Source Software Configuration Management , 2013, 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.

[22]  Sergey Fomel,et al.  Reproducible Computational Experiments using Scons , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[23]  Les Hatton,et al.  Full Computational Reproducibility in Biological Science: Methods, Software and a Case Study in Protein Biology , 2016, 1608.06897.

[24]  Carole A. Goble,et al.  Results May Vary: Reproducibility, Open Science and All That Jazz , 2013, LISC@ISWC.

[25]  Matjaz B. Juric,et al.  Business Process Execution Language for Web Services BPEL and BPEL4WS 2nd Edition , 2006 .

[26]  Doreen Schweizer,et al.  Metal Fatigue Analysis Handbook Practical Problem Solving Techniques For Computer Aided Engineering , 2016 .

[27]  Jeffrey T. Leek,et al.  Opinion: Reproducible research can still be wrong: Adopting a prevention approach , 2015, Proceedings of the National Academy of Sciences.

[28]  Ivar Jacobson,et al.  The unified modeling language reference manual , 2010 .

[29]  Jeffrey G. Gray,et al.  A Graphical Modeling Environment for the Generation of Workflows for the Globus Toolkit , 2005 .

[30]  Olivier Richard,et al.  A survey of general-purpose experiment management tools for distributed systems , 2015, Future Gener. Comput. Syst..

[31]  Arnaud Legrand,et al.  An Effective Git And Org-Mode Based Workflow For Reproducible Research , 2015, OPSR.

[32]  M. Ragan-Kelley,et al.  The Jupyter/IPython architecture: a unified view of computational research, from interactive exploration to communication and publication. , 2014 .

[33]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[34]  Joseph Emeras,et al.  Reconstructing the software environment of an experiment with kameleon , 2012, COMPUTE.

[35]  Kaizar Amin,et al.  GridAnt: a client-controllable grid workflow system , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[36]  Yuriy Brun,et al.  Supporting Undo and Redo in Scientific Data Analysis , 2013, TaPP.

[37]  Christopher Olston,et al.  Stateful bulk processing for incremental analytics , 2010, SoCC '10.

[38]  Joyce M. Ray,et al.  Research Data Management: Practical Strategies for Information Professionals , 2014 .

[39]  Eelco Visser,et al.  Automated software testing and release with nix build farms , 2007 .

[40]  Shigeo Kawata,et al.  Computer-Assisted Simulation Environment for Partial-Differential-Equation(PDE) Problems : 1. PDE Problem Description and Program Generation , 1997 .

[41]  J. Crocker,et al.  Addressing Scientific Fraud , 2011, Science.

[42]  Andrew P. Davison.  Automated Capture of Experiment Context for Easier Reproducibility in Computational Research , 2012, Computing in Science & Engineering.

[43]  Seth Vargo,et al.  Learning Chef: A Guide to Configuration Management and Automation , 2013 .

[44]  Marlon Dumas,et al.  UML Activity Diagrams as a Workflow Specification Language , 2001, UML.

[45]  Wil M.P. van der Aalst,et al.  YAWL: yet another workflow language , 2005, Inf. Syst..

[46]  Jeffrey G. Gray,et al.  Grid‐Flow: a Grid‐enabled scientific workflow system with a Petri‐net‐based interface , 2006, Concurr. Comput. Pract. Exp..

[47]  Chris R. Johnson.  Top Scientific Visualization Research Problems , 2004, IEEE Computer Graphics and Applications.

[48]  Dror G. Feitelson,et al.  From Repeatability to Reproducibility and Corroboration , 2015, OPSR.

[49]  Omar S. Gómez,et al.  Replication , Reproduction and Re-analysis : Three ways for verifying experimental , 2010 .

[50]  Steven Loria,et al.  Collaborating with Git , 2016 .

[51]  Frank Leymann.  Web Services and Their Composition , 2001, EWSPT.

[52]  Paul Sava,et al.  Madagascar: open-source software project for multidimensional data analysis and reproducible computational experiments , 2013 .

[53]  Yolanda Gil,et al.  Pegasus: Mapping Scientific Workflows onto the Grid , 2004, European Across Grids Conference.

[54]  Matthew J. Turk,et al.  Scaling a code in the human dimension , 2013, XSEDE.

[55]  Sergey Fomel,et al.  Reproducible Research as a Community Effort: Lessons from the Madagascar Project , 2015, Computing in Science & Engineering.

[56]  Cláudio T. Silva,et al.  VisTrails: enabling interactive multiple-view visualizations , 2005, VIS 05. IEEE Visualization, 2005..

[57]  Predrag Buncic,et al.  CernVM-FS: delivering scientific software to globally distributed computing resources , 2011, NDM '11.

[58]  Steven Knight,et al.  Building software with SCons , 2005, Comput. Sci. Eng..

[59]  Dennis Shasha,et al.  ReproZip: Using Provenance to Support Computational Reproducibility , 2013, TaPP.

[60]  Hadas Kress-Gazit,et al.  Computer-Assisted Engineering for Robotics and Autonomous Systems (Dagstuhl Seminar 17071) , 2017, Dagstuhl Reports.

[61]  Jon F. Claerbout,et al.  Electronic documents give reproducible research a new meaning: 62nd Ann , 1992 .

[62]  Jim Q. Smith,et al.  Estimating Causal Structure Using Conditional DAG Models , 2014, J. Mach. Learn. Res..

[63]  Robert K. Cunningham,et al.  Automated Assessment of Secure Search Systems , 2015, OPSR.

[64]  Sorin Lerner,et al.  OPIUM: Optimal Package Install/Uninstall Manager , 2007, 29th International Conference on Software Engineering (ICSE'07).

[65]  Yolanda Gil,et al.  Provenance trails in the Wings-Pegasus system , 2008 .

[66]  C. M. Sperberg-McQueen,et al.  Extensible Markup Language (XML) , 1997, World Wide Web J..

[67]  Bertram Ludäscher,et al.  Kepler: an extensible system for design and execution of scientific workflows , 2004, Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004..

[68]  Daniel J. Blankenberg,et al.  Galaxy: a platform for interactive large-scale genome analysis. , 2005, Genome research.

[69]  Roger D. Peng,et al.  The reproducibility crisis in science: A statistical counterattack , 2015 .

[70]  Jonathan M. Borwein,et al.  SIAM: “Setting the Default to Reproducible” in Computational Science Research , 2013 .

[71]  GilYolanda,et al.  Provenance trails in the Wings-Pegasus system , 2008 .

[72]  Suzanne J. Matthews,et al.  Paper Mâché: Creating Dynamic Reproducible Science , 2011, ICCS.

[73]  Bo Einarsson.  Accuracy and Reliability in Scientific Computing (Software, Environments, Tools) (Software, Environments, Tools) , 2005 .

[74]  Marta Mattoso,et al.  A Survey of Data-Intensive Scientific Workflow Management , 2015, Journal of Grid Computing.

[75]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[76]  Joseph Emeras,et al.  Reproducible Software Appliances for Experimentation , 2014, TRIDENTCOM.

[77]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[78]  Marco Torchiano,et al.  On the effectiveness of the test-first approach to programming , 2005, IEEE Transactions on Software Engineering.

[79]  Ian M. Mitchell,et al.  Best Practices for Scientific Computing , 2012, PLoS biology.

[80]  Niky Riga,et al.  Creating Repeatable Computer Science and Networking Experiments on Shared, Public Testbeds , 2015, OPSR.

[81]  Ronald F. Boisvert,et al.  Uncertainty Quantification in Scientific Computing , 2012, IFIP Advances in Information and Communication Technology.

[82]  Ivar Jacobson,et al.  Unified Modeling Language Reference Manual, The (2nd Edition) , 2004 .

[83]  SchwabMatthias,et al.  Making scientific computations reproducible , 2000 .

[84]  L. Aversano,et al.  FlowManager: a workflow management system based on Petri nets , 2002, Proceedings 26th Annual International Computer Software and Applications.

[85]  Tim Brecht,et al.  Conducting Repeatable Experiments and Fair Comparisons using 802.11n MIMO Networks , 2015, OPSR.

[86]  Alex M. Warren.  Repeatability and Benefaction in Computer Systems Research — A Study and a Modest Proposal , 2015 .

[87]  Cláudio T. Silva,et al.  VisTrails: visualization meets data management , 2006, SIGMOD Conference.

[88]  Ian M. Mitchell,et al.  Reproducible research for scientific computing: Tools and strategies for changing the culture , 2012, Computing in Science & Engineering.

[89]  C. Drummond.  Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[90]  Thomas T. Hewett,et al.  Toward a Human Centered Scientific Problem Solving Environment , 2000 .

[91]  Christian Dietrich,et al.  The dataref versuchung: Saving Time through Better Internal Repeatability , 2015, OPSR.

[92]  Douglas Thain,et al.  Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids , 2015, VTDC@HPDC.

[93]  Shigeo Kawata,et al.  Computer-Assisted Simulation Environment for Partial-Differential-Equation Problems : 2. Visualization and Steering of Program Generation Process , 1997 .

[94]  Bill Howe,et al.  Virtual Appliances, Cloud Computing, and Reproducible Research , 2012, Computing in Science & Engineering.

[95]  J. Brooks.  Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[96]  Duncan Dubugras Alcoba Ruiz,et al.  Extending UML activity diagram for workflow modeling in production systems , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[97]  Molnár Gyöngyvér,et al.  Exploration and learning strategies in an interactive problem-solving environment at the beginning of higher education studies , 2017 .

[98]  Yuqing Wu,et al.  Towards Reproducible eScience in the Cloud , 2011, 2011 IEEE Third International Conference on Cloud Computing Technology and Science.

[99]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[100]  Karl Fogel,et al.  Producing open source software - how to run a successful free software project , 2005 .

[101]  Robert Mecklenburg.  Managing Projects with GNU Make , 2004 .

[102]  Paul Anderson,et al.  Configuration Tools: Working Together , 2005, LISA.

[103]  Howard T. Everson,et al.  Methodology Review: Statistical Approaches for Assessing Measurement Bias , 1993 .

[104]  B. Jasny,et al.  Again, and Again, and Again … , 2011 .

[105]  Wil M. P. van der Aalst,et al.  XRL/Flower: Supporting Inter-organizational Workflows Using XML/Petri-Net Technology , 2002, WES.

[106]  Paolo Di Tommaso,et al.  Using the Nextflow framework for reproducible in-silico omics analyses across clusters and clouds , 2017 .

[107]  C. M. Sperberg-McQueen,et al.  Extensible markup language , 1997 .

[108]  John R. Rice.  Future Challenges for Scientific Simulation , 2000 .

[109]  Michael Chow,et al.  Eidetic Systems , 2014, OSDI.

[110]  Predrag Buncic,et al.  Long-term preservation of analysis software environment , 2012 .

[111]  Sudhir Pandey,et al.  Investigating Community, Reliability and Usability of CFEngine, Chef and Puppet , 2012 .

[112]  Matthias Schwab,et al.  Making scientific computations reproducible , 2000, Comput. Sci. Eng..

[113]  Barry W. Boehm,et al.  Software Risk Management , 1989, ESEC.

[114]  Carole A. Goble,et al.  Taverna: a tool for building and running workflows of services , 2006, Nucleic Acids Res..

[115]  Matthias Schwab,et al.  Empowering SEP's documents , 2000 .

[116]  Ian J. Taylor,et al.  The Triana Workflow Environment: Architecture and Applications , 2007, Workflows for e-Science, Scientific Workflows for Grids.

[117]  W.M.P. van der Aalst,et al.  YAWL: yet another workflow language (revised version) , 2003 .

[118]  Edward A. Lee,et al.  CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. 2000; 00:1–7 Prepared using cpeauth.cls [Version: 2002/09/19 v2.02] Taverna: Lessons in creating , 2022 .

[119]  Victoria Stodden,et al.  Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research , 2014 .

[120]  John Bresnahan,et al.  Managing appliance launches in infrastructure clouds , 2011, TG.

[121]  V. Vianu,et al.  Edinburgh Why and Where: A Characterization of Data Provenance , 2017 .

[122]  Eric A. Brewer,et al.  Kubernetes and the path to cloud native , 2015, SoCC.

[123]  et al.,et al.  Jupyter Notebooks - a publishing format for reproducible computational workflows , 2016, ELPUB.

[124]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[125]  Shigeo Kawata,et al.  Scientific Simulation Execution Support on a Closed Distributed Computer Environment , 2006, 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06).

[126]  Shigeo Kawata,et al.  A Distributed Education-Support PSE System , 2007, Third IEEE International Conference on e-Science and Grid Computing (e-Science 2007).

[127]  Bill Howe,et al.  CDE: A Tool for Creating Portable Experimental Software Packages , 2012 .

[128]  Wei Chen,et al.  Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification , 2012 .

[129]  Jill P Mesirov,et al.  Accessible Reproducible Research , 2010, Science.

[130]  Yolanda Gil,et al.  Detecting common scientific workflow fragments using templates and execution provenance , 2013, K-CAP.

[131]  Kevin Murrell.  The Harwell Dekatron Computer , 2013, HC.

[132]  Wil vanderAalst,et al.  Workflow Management: Models, Methods, and Systems , 2004 .

[133]  Andrea C. Arpaci-Dusseau,et al.  The Popper Convention: Making Reproducible Systems Evaluation Practical , 2017, 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW).

[134]  Paul T. Groth,et al.  Ten Simple Rules for the Care and Feeding of Scientific Data , 2014, PLoS Comput. Biol..

[135]  K. Mani Chandy,et al.  Managing Specificity and Generality: Tailoring General Archetypal PSEs to Specific Users , 2000 .

[136]  Charles Zheng,et al.  An analysis of reproducibility and non-determinism in HEP software and ROOT data , 2017 .

[137]  Yogesh L. Simmhan,et al.  A survey of data provenance in e-science , 2005, SGMD.

[138]  Andres Löh,et al.  NixOS: A purely functional Linux distribution , 2010, J. Funct. Program..

[139]  Ben Marwick,et al.  Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation , 2016, Journal of Archaeological Method and Theory.

[140]  Yuan Guo-xing,et al.  Verification and Validation in Scientific Computing Code , 2010 .

[141]  Victoria Stodden,et al.  Implementing Reproducible Research , 2018 .

[142]  Bertram Ludäscher,et al.  Compiling abstract scientific workflows into Web service workflows , 2003, 15th International Conference on Scientific and Statistical Database Management, 2003..

[143]  John R. Rice,et al.  Enabling Technologies for Computational Science: Frameworks, Middleware and Environments , 2012 .

[144]  Luc Moreau,et al.  Recording and Reasoning over Data Provenance in Web and Grid Services , 2003, OTM.

[145]  Emmanuel Jeanvoine,et al.  Kadeploy3: Efficient and Scalable Operating System Provisioning for HPC Clusters , 2012 .

[146]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[147]  Barbara Sierman,et al.  The SCAPE Policy Framework, maturity levels and the need for realistic preservation policies , 2014, iPRES.

[148]  Matjaz B. Juric,et al.  Business process execution language for web services , 2004 .

[149]  E. Gallopoulos,et al.  Computer as thinker/doer: problem-solving environments for computational science , 1994, IEEE Computational Science and Engineering.

[150]  Rajkumar Buyya,et al.  A novel architecture for realizing grid workflow using tuple spaces , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[151]  Boisvert,et al.  Accuracy and Reliability in Scientific Computing , 2006 .

[152]  R. Bell,et al.  The million dollar programming prize , 2009, IEEE Spectrum.

[153]  James Loope.  Managing Infrastructure with Puppet , 2011 .

[154]  Andreas Hoheisel,et al.  User tools and languages for graph‐based Grid workflows , 2006, Concurr. Comput. Pract. Exp..

[155]  Gregor von Laszewski,et al.  Java CoG Kit Workflow , 2007, Workflows for e-Science, Scientific Workflows for Grids.

[156]  Bruce W. Char,et al.  Workshop on Scientific Knowledge, Information and Computing (SIDEKIC 98) , 2000 .

[157]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[158]  Cory Lueninghoener.  Getting Started with Configuration Management , 2011, login Usenix Mag..

[159]  Michela Taufer,et al.  On the Need for Reproducible Numerical Accuracy through Intelligent Runtime Selection of Reduction Algorithms at the Extreme Scale , 2015, 2015 IEEE International Conference on Cluster Computing.

[160]  Carsten Dominik,et al.  The Org Mode 7 Reference Manual - Organize your life with GNU Emacs , 2010 .

[161]  Ludovic Courtès,et al.  Reproducible and User-Controlled Software Environments in HPC with Guix , 2015, Euro-Par Workshops.

[162]  Thain Douglas,et al.  PRUNE: A preserving run environment for reproducible scientific computing , 2016 .

[163]  Franck Cappello,et al.  Grid'5000: a large scale and highly reconfigurable grid experimental testbed , 2005, The 6th IEEE/ACM International Workshop on Grid Computing, 2005..

[164]  Wil M. P. van der Aalst,et al.  The Application of Petri Nets to Workflow Management , 1998, J. Circuits Syst. Comput..

[165]  Lenin Ravindranath,et al.  Nectar: Automatic Management of Data and Computation in Datacenters , 2010, OSDI.

[166]  Reagan Moore,et al.  iRODS Primer: Integrated Rule-Oriented Data System , 2010, iRODS Primer.

[167]  David L. Donoho,et al.  WaveLab and Reproducible Research , 1995 .

[168]  Olivier Beaumont,et al.  Aquarels: A Problem-Solving Environment for Validating Scientific Software , 2000 .

[169]  S. Goodman,et al.  Reproducible Research: Moving toward Research the Public Can Really Trust , 2007, Annals of Internal Medicine.

[170]  Shigeo Kawata,et al.  Computer Assisted Problem Solving Environment (PSE) , 2015 .

[171]  Cláudio T. Silva,et al.  Provenance for Computational Tasks: A Survey , 2008, Computing in Science & Engineering.