The case for open computer programs

Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.

[1]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[2]  David L Donoho,et al.  An invitation to reproducible computational research. , 2010, Biostatistics.

[3]  J. Gerring.  A case study , 2011, Technology and Society.

[4]  Bruce Alberts,et al.  Making Data Maximally Available , 2011, Science.

[5]  Ted Pedersen,et al.  Empiricism Is Not a Matter of Faith , 2008, Computational Linguistics.

[6]  Barry Boehm,et al.  Foundations of Empirical Software Engineering , 2005 .

[7]  P. Jones,et al.  Uncertainty estimates in regional and global observed temperature changes: A new data set from 1850 , 2006 .

[8]  Kees van Deemter.  Not Exactly: In Praise of Vagueness , 2010 .

[9]  Peter Linz,et al.  Accurate floating-point summation , 1970, CACM.

[10]  T. R. Anderson,et al.  Medusa-1.0: a new intermediate complexity plankton ecosystem model for the global domain , 2010 .

[11]  Jean-Michel Muller,et al.  Exact and Approximated Error of the FMA , 2011, IEEE Transactions on Computers.

[12]  Nick Barnes.  Publish your computer code: it is good enough , 2010, Nature.

[13]  Tony Hey,et al.  The Fourth Paradigm: Data-Intensive Scientific Discovery , 2009 .

[14]  Steve M. Easterbrook,et al.  Engineering the Software for Understanding Climate Change , 2009, Computing in Science & Engineering.

[15]  Chris H. Q. Ding,et al.  Using accurate arithmetics to improve numerical reproducibility and stability in parallel applications , 2000, ICS '00.

[16]  Nathalie Revol.  Standardized Interval Arithmetic and Interval Arithmetic Used in Libraries , 2010, ICMS.

[17]  Jill P Mesirov,et al.  Accessible Reproducible Research , 2010, Science.

[18]  Eric S. Raymond,et al.  The cathedral and the bazaar , 1998, First Monday.

[19]  Arian Maleki,et al.  Reproducible Research in Computational Harmonic Analysis , 2009, Computing in Science & Engineering.

[20]  Meine van der Meulen,et al.  The Effectiveness of Software Diversity in a Large Population of Programs , 2008, IEEE Transactions on Software Engineering.

[21]  Alexandru Nicolau,et al.  Improving accuracy for matrix multiplications on GPUs , 2011, Sci. Program..

[22]  Matthias Schwab,et al.  Making scientific computations reproducible , 2000, Comput. Sci. Eng..

[23]  Fabian de Bruijn,et al.  Ambiguity in Natural Language Software Requirements: A Case Study , 2010, REFSQ.

[24]  Robert Laws,et al.  The seismic kernel system — a large‐scale exercise in fortran 77 portability , 1988, Softw. Pract. Exp..

[25]  Stephen J. Thomas,et al.  An Ensemble Analysis of Forecast Errors Related to Floating Point Performance , 2002 .

[26]  Didar Zowghi,et al.  On the Role of Ambiguity in RE , 2010, REFSQ.

[27]  Dennis McCafferty.  Should code be released? , 2010, Commun. ACM.

[28]  Siegfried M. Rump.  Accurate and Reliable Computing in Floating-Point Arithmetic , 2010, ICMS.

[29]  Z. Merali.  Computational science: ...Error , 2010, Nature.

[30]  Siegfried M. Rump,et al.  Accurate Floating-Point Summation Part I: Faithful Rounding , 2008, SIAM J. Sci. Comput..

[31]  Paul McKellips,et al.  Good enough , 2013, Lab Animal.

[32]  Bashar Nuseibeh,et al.  Automatic detection of nocuous coordination ambiguities in natural language requirements , 2010, ASE '10.

[33]  Victor Y. Pan,et al.  A new error-free floating-point summation algorithm , 2009, Comput. Math. Appl..

[34]  Les Hatton,et al.  The T-experiments: errors in scientific software , 1996, Quality of Numerical Software.

[35]  Edward N. Adams,et al.  Optimizing Preventive Service of Software Products , 1984, IBM J. Res. Dev..

[36]  Shari Lawrence Pfleeger,et al.  Investigating the Influence of Formal Methods , 1997, Computer.

[37]  Andy Roberts,et al.  How Accurate Is Scientific Software? , 1994, IEEE Trans. Software Eng..