Property-based Sensitivity Analysis: An approach to identify model implementation and integration errors

[1]  Xi Chen,et al.  Uncertainty analysis of a semi-distributed hydrologic model based on a Gaussian Process emulator , 2018, Environ. Model. Softw..

[2]  S. Uhlenbrook,et al.  Sensitivity analyses of a distributed catchment model to verify the model structure , 2005 .

[3]  Walter Rudametkin,et al.  An approach and benchmark to detect behavioral changes of commits in continuous integration , 2020, Empirical Software Engineering.

[4]  José Ferrer,et al.  An improved sampling strategy based on trajectory design for application of the Morris method to systems with many input factors , 2012, Environ. Model. Softw..

[5]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[6]  Sander Janssen,et al.  Evaluating OpenMI as a model integration platform across disciplines , 2013, Environ. Model. Softw..

[7]  Anthony J. Jakeman,et al.  A tiered, system-of-systems modeling framework for resolving complex socio-environmental policy issues , 2019, Environ. Model. Softw..

[8]  Francesca Pianosi,et al.  A simple and efficient method for global sensitivity analysis based on cumulative distribution functions , 2015, Environ. Model. Softw..

[9]  William Becker,et al.  Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices , 2017, Environ. Model. Softw..

[10]  Steven F Railsback,et al.  Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[11]  Francesca Pianosi,et al.  Global Sensitivity Analysis of environmental models: Convergence and validation , 2016, Environ. Model. Softw..

[12]  Michiru Nagatsu,et al.  What does interdisciplinarity look like in practice: Mapping interdisciplinarity and its limits in the environmental sciences. , 2018, Studies in history and philosophy of science.

[13]  Francesca Pianosi,et al.  Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model , 2017, Environ. Model. Softw..

[14]  J. Norton.  Selection of Morris Trajectories for Initial Sensitivity Analysis , 2009 .

[15]  Roger David Braddock,et al.  Application of the Morris algorithm for sensitivity analysis of the REALM model for the Goulburn irrigation system , 2006 .

[16]  Will Usher,et al.  SALib: An open-source Python library for Sensitivity Analysis , 2017, J. Open Source Softw..

[17]  Jennifer Badham,et al.  Socio-technical scales in socio-environmental modeling: Managing a system-of-systems modeling approach , 2020, Environmental Modelling & Software.

[18]  Rolf Hut,et al.  Let hydrologists learn the latest computer science by working with Research Software Engineers (RSEs) and not reinvent the waterwheel ourselves. A comment to “Most Computational Hydrology is not Reproducible, so is it Really Science?” , 2017 .

[19]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[20]  Paola Annoni,et al.  Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index , 2010, Comput. Phys. Commun..

[21]  H Rabitz,et al.  Systems Analysis at the Molecular Scale , 1989, Science.

[22]  W. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[23]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[24]  Paula A. Harrison,et al.  Exploring scenario and model uncertainty in cross-sectoral integrated assessment approaches to climate change impacts , 2014, Climatic Change.

[25]  Philip G Williams,et al.  Characterization of Leptazolines A-D, Polar Oxazolines from the Cyanobacterium Leptolyngbya sp., Reveals a Glitch with the "Willoughby-Hoye" Scripts for Calculating NMR Chemical Shifts. , 2019, Organic letters.

[26]  Richard C. Gerkin,et al.  Unit testing, model validation, and biological simulation , 2015, F1000Research.

[27]  Patrick M. Reed,et al.  Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models , 2013 .

[28]  Abbas Sharifi,et al.  Modeling and sensitivity analysis of NOx emissions and mechanical efficiency for diesel engine , 2019, Environmental Science and Pollution Research.

[29]  Konstantinos Sagonas,et al.  Targeted property-based testing , 2017, ISSTA.

[30]  Emanuele Borgonovo,et al.  Sensitivity analysis with finite changes: An application to modified EOQ models , 2010, Eur. J. Oper. Res..

[31]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[32]  Anthony J. Jakeman,et al.  Selecting among five common modelling approaches for integrated environmental assessment and management , 2013, Environ. Model. Softw..

[33]  Sabine Attinger,et al.  Computationally inexpensive identification of noninformative model parameters by sequential screening , 2015 .

[34]  Francesca Pianosi,et al.  What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling , 2019 .

[35]  Anthony J. Jakeman,et al.  Development of an integrated model for the Campaspe catchment: a tool to help improve understanding of the interaction between society, policy, farming decision, ecology, hydrology and climate , 2018, Proceedings of the International Association of Hydrological Sciences.

[36]  Carole A. Goble,et al.  The Software Sustainability Institute: Changing Research Software Attitudes and Practices , 2013, Computing in Science & Engineering.

[37]  Stefano Tarantola,et al.  Trends in sensitivity analysis practice in the last decade. , 2016, The Science of the total environment.

[38]  Uta Berger,et al.  Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology , 2005, Science.

[39]  Laura Christopherson,et al.  Water Science Software Institute: Agile and Open Source Scientific Software Development , 2014, Computing in Science & Engineering.

[40]  Herschel Rabitz,et al.  Sixth International Conference on Sensitivity Analysis of Model Output Global Sensitivity Analysis for Systems with Independent and / or Correlated Inputs , 2013 .

[41]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[42]  M.J.R. Knapen,et al.  An IT perspective on integrated environmental modelling: The SIAT case , 2010 .

[43]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[44]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[45]  Alexey A. Voinov,et al.  'Integronsters', integral and integrated modeling , 2013, Environ. Model. Softw..

[46]  Jing Yang,et al.  Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis , 2011, Environ. Model. Softw..

[47]  Andrea Saltelli,et al.  A sensitivity analysis of the PAWN sensitivity index , 2019, Environ. Model. Softw..

[48]  B Boehm.  A spiral model of software development and enhancement , 1986, SOEN.

[49]  James M. Bieman,et al.  Testing scientific software: A systematic literature review , 2014, Inf. Softw. Technol..

[50]  X. Y. Sun,et al.  Three complementary methods for sensitivity analysis of a water quality model , 2012, Environ. Model. Softw..

[51]  Joseph H. A. Guillaume,et al.  Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose , 2019, Environ. Model. Softw..

[52]  Deana McDonagh,et al.  Shared language:Towards more effective communication. , 2013, The Australasian medical journal.

[53]  ElSawahSondoss,et al.  Integrated assessment and modelling , 2015 .

[54]  Keith Beven,et al.  Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology , 2001 .

[55]  A. Saltelli,et al.  Making best use of model evaluations to compute sensitivity indices , 2002 .

[56]  Takuya Iwanaga,et al.  A socio-environmental model for exploring sustainable water management futures: Participatory and collaborative modelling in the Lower Campaspe catchment , 2020 .

[57]  B. Croke,et al.  A review of foundational methods for checking the structural identifiability of models: Results for rainfall-runoff , 2015 .

[58]  Janice Singer,et al.  How do scientists develop and use scientific software? , 2009, 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering.

[59]  Christopher Hutton,et al.  Most computational hydrology is not reproducible, so is it really science? , 2016, Water Resources Research.

[60]  Andrea Saltelli,et al.  From screening to quantitative sensitivity analysis. A unified approach , 2011, Comput. Phys. Commun..

[61]  Raymond Reiter,et al.  A Theory of Diagnosis from First Principles , 1986, Artif. Intell..

[62]  Steve Easterbrook,et al.  Open code for open science , 2014 .

[63]  Takuya Iwanaga,et al.  Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques , 2020, Environ. Model. Softw..

[64]  James Spelling,et al.  Benchmarking the PAWN distribution-based method against the variance-based method in global sensitivity analysis: Empirical results , 2019, Environ. Model. Softw..

[65]  Emanuele Borgonovo,et al.  Global sensitivity measures from given data , 2013, Eur. J. Oper. Res..

[66]  Marc B. Neumann,et al.  Global sensitivity analysis for urban water quality modelling: terminology, convergence and comparison of different methods. , 2015 .

[67]  Casper Lassenius,et al.  Problems, causes and solutions when adopting continuous delivery - A systematic literature review , 2017, Inf. Softw. Technol..

[68]  Francesca Pianosi,et al.  Distribution-based sensitivity analysis from a generic input-output sample , 2018, Environ. Model. Softw..

[69]  D. Hamby.  A review of techniques for parameter sensitivity analysis of environmental models , 1994, Environmental monitoring and assessment.

[70]  Ben Marwick,et al.  Truth, Proof, and Reproducibility: There’s No Counter-Attack for the Codeless , 2019, Communications in Computer and Information Science.

[71]  Herschel Rabitz,et al.  Global uncertainty assessments by high dimensional model representations (HDMR) , 2002 .

[72]  Yuqiong Liu,et al.  Reconciling theory with observations: elements of a diagnostic approach to model evaluation , 2008 .

[73]  Amr Mossalam.  Projects’ issue management , 2018 .

[74]  Saman Razavi,et al.  What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models , 2015 .

[75]  Matt Bishop,et al.  Property-based testing: a new approach to testing for assurance , 1997, SOEN.

[76]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[77]  Emanuele Borgonovo,et al.  The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support , 2020, Environ. Model. Softw..

[78]  William Samuelson,et al.  Status quo bias in decision making , 1988 .

[79]  Saman Razavi,et al.  VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis , 2019, Environ. Model. Softw..

[80]  Elmar Plischke,et al.  An effective algorithm for computing global sensitivity indices (EASI) , 2010, Reliab. Eng. Syst. Saf..