Truth, Proof, and Reproducibility: There’s No Counter-Attack for the Codeless

Current concerns about reproducibility in many research communities can be traced back to a high value placed on empirical reproducibility of the physical details of scientific experiments and observations. For example, the detailed descriptions by 17th century scientist Robert Boyle of his vacuum pump experiments are often held to be the ideal of reproducibility as a cornerstone of scientific practice. Victoria Stodden has claimed that the computer is an analog for Boyle's pump -- another kind of scientific instrument that needs detailed descriptions of how it generates results. In the place of Boyle's hand-written notes, we now expect code in open source programming languages to be available to enable others to reproduce and extend computational experiments. In this paper we show that there is another genealogy for reproducibility, starting at least from Euclid, in the production of proofs in mathematics. Proofs have a distinctive quality of being necessarily reproducible, and are the cornerstone of mathematical science. However, the task of the modern mathematical scientist has drifted from that of blackboard rhetorician, where the craft of proof reigned, to a scientific workflow that now more closely resembles that of an experimental scientist. So, what is proof in modern mathematics? And, if proof is unattainable in other fields, what is due scientific diligence in a computational experimental environment? How do we measure truth in the context of uncertainty? Adopting a manner of Lakatosian conversant conjecture between two mathematicians, we examine how proof informs our practice of computational statistical inquiry. We propose that a reorientation of mathematical science is necessary so that its reproducibility can be readily assessed.

[1]  Achim Zeileis,et al.  CRAN Task Views , 2015 .

[2]  R. Peng Reproducible Research in Computational Science , 2011, Science.

[3]  David L Donoho,et al.  An invitation to reproducible computational research. , 2010, Biostatistics.

[4]  Brian A. Davey,et al.  The Homomorphism Lattice Induced by a Finite Algebra , 2016, Order.

[5]  Ian M. Mitchell,et al.  Reproducible research for scientific computing: Tools and strategies for changing the culture , 2012, Computing in Science & Engineering.

[6]  R. Lanfear,et al.  The Extent and Consequences of P-Hacking in Science , 2015, PLoS biology.

[7]  Brian A. Davey,et al.  Introduction to Lattices and Order: Frontmatter , 2002 .

[8]  M. Sørensen,et al.  Lectures on the Curry-Howard Isomorphism , 2013 .

[9]  Brian A. Davey,et al.  An Introduction to Lattices and Order , 1989 .

[10]  Francisco Rodríguez-Sánchez,et al.  Ciencia reproducible: qué, por qué, cómo , 2016 .

[11]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[12]  Morton N. Cohen,et al.  The Annotated Alice: The Definitive Edition (review) , 2001 .

[13]  A. Pickering The mangle of practice : time, agency, and science , 1997 .

[14]  F. Fidler,et al.  Questionable research practices in ecology and evolution , 2018, PloS one.

[15]  Sander Greenland,et al.  Scientists rise up against statistical significance , 2019, Nature.

[16]  A. Bundy,et al.  What is a proof? , 2005, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[17]  Anna Nowogrodzki,et al.  How to support open-source software and stay sane , 2019, Nature.

[18]  Yves Bertot,et al.  A Short Presentation of Coq , 2008, TPHOLs.

[19]  Ben Marwick,et al.  Packaging Data Analytical Work Reproducibly Using R (and Friends) , 2018 .

[20]  M. Sørensen,et al.  Lectures on the Curry-Howard Isomorphism, Volume 149 (Studies in Logic and the Foundations of Mathematics) , 2006 .

[21]  S. Shapin,et al.  Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life , 1987 .

[22]  J. Ioannidis,et al.  Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017 , 2018, PLoS biology.

[23]  Sara Varela Cómo crear paquetes de R: “R packages. Organize, test, document, and share your code” por Hadley Wickham, 2015 , 2015 .

[24]  Per Martin-Löf,et al.  Constructive mathematics and computer programming , 1984 .

[25]  Hilary S. Parker,et al.  Opinionated analysis development , 2017 .

[26]  R. Merton,et al.  On social structure and science , 1996 .

[27]  Hadley Wickham,et al.  testthat: Get Started with Testing , 2011, R J..

[28]  Ian M. Mitchell,et al.  Best Practices for Scientific Computing , 2012, PLoS biology.

[29]  Jennifer Bryan Excuse Me, Do You Have a Moment to Talk About Version Control? , 2018 .

[30]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[31]  Fiona Fidler,et al.  Reproducibility of Scientific Results , 2018 .

[32]  Benjamin S. Baumer,et al.  Tidy data , 2022, Modern Data Science with R.

[33]  Susan Haack,et al.  Defending Science -- Within Reason : Between Scientism and Cynicism , 2003 .

[34]  Jonathan M. Borwein,et al.  SIAM: “Setting the Default to Reproducible” in Computational Science Research , 2013 .

[35]  Imre Lakatos,et al.  On the Uses of Rigorous Proof. (Book Reviews: Proofs and Refutations. The Logic of Mathematical Discovery) , 1977 .