Keeping modelling notebooks with TRACE: Good for you and good for environmental research and management support

Abstract The acceptance and usefulness of simulation models are often limited by the efficiency, transparency, reproducibility, and reliability of the modelling process. We address these issues by suggesting that modellers (1) “trace” the iterative modelling process by keeping a modelling notebook corresponding to the laboratory notebooks used by empirical researchers, (2) use a standardized notebook structure and terminology based on the existing TRACE documentation framework, and (3) use their notebooks to compile TRACE documents that supplement publications and reports. These practices have benefits for model developers, users, and stakeholders: improved and efficient model design, analysis, testing, and application; increased model acceptance and reuse; and replicability and reproducibility of the model and the simulation experiments. Using TRACE terminology and structure in modelling notebooks facilitates production of TRACE documents. We explain the rationale of TRACE, provide example TRACE documents, and suggest strategies for keeping “TRACE Modelling Notebooks.”

[1]  Kai Blin,et al.  Ten Simple Rules for Taking Advantage of Git and GitHub , 2014, bioRxiv.

[2]  Donald E. Knuth,et al.  Literate Programming , 1984, Comput. J..

[3]  Ning Jiang,et al.  Our path to better science in less time using open data science tools , 2017, Nature Ecology &Evolution.

[4]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[5]  Uta Berger,et al.  From cases to general principles: A call for theory development through agent-based modeling , 2019, Ecological Modelling.

[6]  Anthony J. Jakeman,et al.  An overview of the system dynamics process for integrated modelling of socio-ecological systems: Lessons on good modelling practice from five case studies , 2017, Environ. Model. Softw..

[7]  Christian E Vincenot,et al.  How new concepts become universal scientific approaches: insights from citation network analysis of agent-based complex systems science , 2018, Proceedings of the Royal Society B: Biological Sciences.

[8]  Cyrille Rossant,et al.  Sustainable computational science: the ReScience initiative , 2017, PeerJ Comput. Sci..

[9]  Peer-Olaf Siebers,et al.  'One Size Does Not Fit All': A Roadmap of Purpose-Driven Mixed-Method Pathways for Sensitivity Analysis of Agent-Based Models , 2020, J. Artif. Soc. Soc. Simul..

[10]  Jennifer Badham,et al.  Effective modeling for Integrated Water Resource Management: A guide to contextual practices by phases and steps and future opportunities , 2019, Environ. Model. Softw..

[11]  Andrew P. Davison,et al.  Learning from the Past: Approaches for Reproducibility in Computational Neuroscience , 2013 .

[12]  Amanda Clare,et al.  Replicating complex agent based models, a formidable task , 2017, Environ. Model. Softw..

[13]  Andreas Focks,et al.  Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE , 2014 .

[14]  Martin Kunc,et al.  Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines , 2019, J. Simulation.

[15]  Xiao Feng,et al.  A standard protocol for reporting species distribution models , 2020, Ecography.

[16]  Volker Grimm,et al.  Three questions to ask before using model outputs for decision support , 2020, Nature Communications.

[17]  Matthias Meyer,et al.  Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments , 2011, Computational and Mathematical Organization Theory.

[18]  Samantha Kanza,et al.  Electronic lab notebooks: can they replace paper? , 2017, Journal of Cheminformatics.

[19]  Sami Domisch,et al.  How to make ecological models useful for environmental management , 2019, Ecological Modelling.

[20]  Sabine Attinger,et al.  Cross-disciplinary links in environmental systems science: Current state and claimed needs identified in a meta-review of process models. , 2018, The Science of the total environment.

[21]  R. Peng.  Reproducible Research in Computational Science , 2011, Science.

[22]  J. Gareth Polhill,et al.  The ODD protocol: A review and first update , 2010, Ecological Modelling.

[23]  Ilkay Altintas,et al.  Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks , 2019, PLoS Comput. Biol..

[24]  Colin Ockleford,et al.  Scientific Opinion on good modelling practice in the context of mechanistic effect models for risk assessment of plant protection products , 2014 .

[25]  Birgit Müller,et al.  A standard protocol for describing individual-based and agent-based models , 2006 .

[26]  Uta Berger,et al.  Making Predictions in a Changing World: The Benefits of Individual-Based Ecology , 2014, Bioscience.

[27]  P. J. Kennedy,et al.  BEEHAVE: a systems model of honeybee colony dynamics and foraging to explore multifactorial causes of colony failure , 2014, The Journal of applied ecology.

[28]  Volker Grimm,et al.  Ecological models supporting environmental decision making: a strategy for the future. , 2010, Trends in ecology & evolution.

[29]  Uta Berger,et al.  Robustness analysis: Deconstructing computational models for ecological theory and applications , 2016 .

[30]  Volker Grimm,et al.  Merging validation and evaluation of ecological models to ‘evaludation’: A review of terminology and a practical approach , 2014 .

[31]  Jason T. Nickla,et al.  Proper Laboratory Notebook Practices: Protecting Your Intellectual Property , 2010, Journal of Neuroimmune Pharmacology.

[32]  Jeffrey M. Perkel,et al.  Why Jupyter is data scientists’ computational notebook of choice , 2018, Nature.

[33]  S. H. Bennekou,et al.  Scientific Opinion on the state of the art of Toxicokinetic/Toxicodynamic (TKTD) effect models for regulatory risk assessment of pesticides for aquatic organisms , 2018, EFSA journal. European Food Safety Authority.

[34]  Bruce Edmonds,et al.  The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism , 2020, J. Artif. Soc. Soc. Simul..

[35]  Ulrich Dirnagl,et al.  A pocket guide to electronic laboratory notebooks in the academic life sciences , 2016, F1000Research.

[36]  Marcin Milkowski,et al.  Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail , 2018, Journal of Computational Neuroscience.

[37]  Robert E. Shannon,et al.  Design and analysis of simulation experiments , 1978, WSC '78.

[38]  King C. Lee.  Good Laboratory Notebook Practices , 2003 .