Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques

Abstract Uncertainty and sensitivity analysis (UA/SA) aid in assessing whether model complexity is warranted and under what conditions. To support these analyses a variety of software tools have been developed to provide UA/SA methods and approaches in a more accessible manner. This paper applies a hybrid bibliometric approach using 11 625 publications sourced from the Web of Science database to identify software packages for UA/SA used within the environmental sciences and to synthesize evidence of general research trends and directions. Use of local sensitivity approaches was determined to be prevalent, although adoption of global sensitivity analysis approaches is increasing. We find that interest in uncertainty management is also increasing, particularly in improving the reliability and effectiveness of UA/SA. Although available software is typically open-source and freely available, uptake of software tools is apparently slow or their use is otherwise under-reported. Longevity is also an issue, with many of the identified software appearing to be unmaintained. Improving the general usability and accessibility of UA/SA tools may help to increase software longevity and the awareness and adoption of purpose-appropriate methods. Usability should be improved so as to lower the "cost of adoption" of incorporating the software in the modelling workflow. An overview of available software is provided to aid modelers in choosing an appropriate software tool for their purposes. Code and representative data used for this analysis can be found at https://github.com/frog7/uasa-trends (10.5281/zenodo.3406946).

[1]  Jan H. Kwakkel,et al.  Exploratory Modeling and Analysis, an approach for model-based foresight under deep uncertainty , 2013 .

[2]  Sergei S. Kucherenko,et al.  Derivative based global sensitivity measures and their link with global sensitivity indices , 2009, Math. Comput. Simul..

[3]  Jon C. Helton,et al.  Multiple predictor smoothing methods for sensitivity analysis: Description of techniques , 2008, Reliab. Eng. Syst. Saf..

[4]  A. Saltelli,et al.  On the Relative Importance of Input Factors in Mathematical Models , 2002 .

[5]  X. Y. Sun,et al.  Three complementary methods for sensitivity analysis of a water quality model , 2012, Environ. Model. Softw..

[6]  Timon Rabczuk,et al.  A software framework for probabilistic sensitivity analysis for computationally expensive models , 2016, Adv. Eng. Softw..

[7]  Patrick M. Reed,et al.  An open source framework for many-objective robust decision making , 2015, Environ. Model. Softw..

[8]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[9]  B. Croke,et al.  Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R , 2013 .

[10]  Evgeny Burnaev,et al.  GTApprox: Surrogate modeling for industrial design , 2016, Adv. Eng. Softw..

[11]  Herschel Rabitz,et al.  Efficient Implementation of High Dimensional Model Representations , 2001 .

[12]  Saltelli Andrea,et al.  Sensitivity Analysis for Nonlinear Mathematical Models. Numerical ExperienceSensitivity Analysis for Nonlinear Mathematical Models. Numerical Experience , 1995 .

[13]  A. Saltelli,et al.  Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques , 1990 .

[14]  Jing Wang,et al.  Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method , 2013, Environ. Model. Softw..

[15]  Hui Zhang,et al.  Experimental explorations on short text topic mining between LDA and NMF based Schemes , 2019, Knowl. Based Syst..

[16]  Martin J. Westgate,et al.  Predicting the time needed for environmental systematic reviews and systematic maps , 2018, Conservation biology : the journal of the Society for Conservation Biology.

[17]  Mufti Mahmud,et al.  A flexible keyphrase extraction technique for academic literature , 2018 .

[18]  Geoffrey T. Parks,et al.  Effective-Quadratures (EQ): Polynomials for Computational Engineering Studies , 2017, J. Open Source Softw..

[19]  Blake Hough,et al.  savvy: visualize high dimensionality sensitivity analysis data. Updated with full sensitivity analysis from ligpy model. , 2016 .

[20]  Bela Gipp,et al.  Research-paper recommender systems: a literature survey , 2015, International Journal on Digital Libraries.

[21]  Lutz Bornmann,et al.  Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references , 2014, J. Assoc. Inf. Sci. Technol..

[22]  Konrad P. Körding,et al.  Science Concierge: A Fast Content-Based Recommendation System for Scientific Publications , 2016, PloS one.

[23]  G. Gary Wang,et al.  Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions , 2010 .

[24]  William Becker,et al.  Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices , 2017, Environ. Model. Softw..

[25]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application , 2016 .

[26]  Prabhat,et al.  Cataloging the visible universe through Bayesian inference in Julia at petascale , 2019, J. Parallel Distributed Comput..

[27]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[28]  Stefano Marelli,et al.  UQLab: a framework for uncertainty quantification in MATLAB , 2014 .

[29]  Wei Gong,et al.  A GUI platform for uncertainty quantification of complex dynamical models , 2016, Environ. Model. Softw..

[30]  Kurt Fedra.  Environmental Modeling Under Uncertainty: Monte Carlo Simulation , 1982 .

[31]  O. Sigmund,et al.  Topology optimization approaches , 2013, Structural and Multidisciplinary Optimization.

[32]  Michael McLennan,et al.  PUQ: A code for non-intrusive uncertainty propagation in computer simulations , 2015, Comput. Phys. Commun..

[33]  G. Deman,et al.  Sensitivity analysis approaches to high-dimensional screening problems at low sample size , 2018 .

[34]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[35]  Sanjeev Arora,et al.  Learning Topic Models -- Going beyond SVD , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[36]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[37]  Stefano Tarantola,et al.  Trends in sensitivity analysis practice in the last decade. , 2016, The Science of the total environment.

[38]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[39]  John Norton,et al.  An introduction to sensitivity assessment of simulation models , 2015, Environ. Model. Softw..

[40]  Emanuele Borgonovo,et al.  Sampling strategies in density-based sensitivity analysis , 2012, Environ. Model. Softw..

[41]  Saman Razavi,et al.  VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis , 2019, Environ. Model. Softw..

[42]  Francesca Pianosi,et al.  A Matlab toolbox for Global Sensitivity Analysis , 2015, Environ. Model. Softw..

[43]  Leonhard Held,et al.  Sensitivity analysis for Bayesian hierarchical models , 2013, 1312.4797.

[44]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[45]  Jing Yang,et al.  Comparing uncertainty analysis techniques for a SWAT application to the Chaohe Basin in China , 2008 .

[46]  Shinichi Nakagawa,et al.  Research Weaving: Visualizing the Future of Research Synthesis. , 2018, Trends in ecology & evolution.

[47]  Peter A. Vanrolleghem,et al.  Uncertainty in the environmental modelling process - A framework and guidance , 2007, Environ. Model. Softw..

[48]  Nick Cramer,et al.  Automatic Keyword Extraction from Individual Documents , 2010 .

[49]  Michael Obersteiner,et al.  Practice and perspectives in the validation of resource management models , 2018, Nature Communications.

[50]  Saman Razavi,et al.  What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models , 2015 .

[51]  Andrea Saltelli,et al.  From screening to quantitative sensitivity analysis. A unified approach , 2011, Comput. Phys. Commun..

[52]  Johan Larsson,et al.  Exploiting active subspaces to quantify uncertainty in the numerical simulation of the HyShot II scramjet , 2014, J. Comput. Phys..

[53]  David B. Lindenmayer,et al.  Software support for environmental evidence synthesis , 2018, Nature Ecology & Evolution.

[54]  Olivier P. Le Maître,et al.  Polynomial chaos expansion for sensitivity analysis , 2009, Reliab. Eng. Syst. Saf..

[55]  Wei Chu,et al.  A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model , 2014, Environ. Model. Softw..

[56]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[57]  L. Shawn Matott,et al.  Evaluating uncertainty in integrated environmental models: A review of concepts and tools , 2009 .

[58]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[59]  Veronica Czitrom,et al.  One-Factor-at-a-Time versus Designed Experiments , 1999 .

[60]  Lior Rokach,et al.  Ensemble learning: A survey , 2018, WIREs Data Mining Knowl. Discov..

[61]  Hans Petter Langtangen,et al.  Chaospy: An open source tool for designing methods of uncertainty quantification , 2015, J. Comput. Sci..

[62]  N. Metropolis,et al.  The Monte Carlo method. , 1949 .

[63]  Mary C. Hill,et al.  UCODE, a computer code for universal inverse modeling 1 Code available at http://water.usgs.gov/soft , 1999 .

[64]  Joseph H. A. Guillaume,et al.  Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose , 2019, Environ. Model. Softw..

[65]  M. Ratto,et al.  Using recursive algorithms for the efficient identification of smoothing spline ANOVA models , 2010 .

[66]  I. Sobol,et al.  About the use of rank transformation in sensitivity analysis of model output , 1995 .

[67]  Martin J. Westgate,et al.  The difficulties of systematic reviews , 2017, Conservation biology : the journal of the Society for Conservation Biology.

[68]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[69]  J. C. Helton,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 1988 .

[70]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory , 2016 .

[71]  Bruno Sudret,et al.  Adaptive sparse polynomial chaos expansion based on least angle regression , 2011, J. Comput. Phys..

[72]  Christopher J. Roy,et al.  A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing , 2011 .

[73]  Sergey Oladyshkin,et al.  Data-driven uncertainty quantification using the arbitrary polynomial chaos expansion , 2012, Reliab. Eng. Syst. Saf..

[74]  D. Kavetski,et al.  Towards a Bayesian total error analysis of conceptual rainfall-runoff models: Characterising model error using storm-dependent parameters , 2006 .

[75]  David Moens,et al.  A survey of non-probabilistic uncertainty treatment in finite element analysis , 2005 .

[76]  T. Ishigami,et al.  An importance quantification technique in uncertainty analysis for computer models , 1990, [1990] Proceedings. First International Symposium on Uncertainty Modeling and Analysis.

[77]  Will Usher,et al.  SALib: An open-source Python library for Sensitivity Analysis , 2017, J. Open Source Softw..

[78]  Thorsten Wagener,et al.  Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox , 2007, Environ. Model. Softw..

[79]  Edward E. Leamer,et al.  Sensitivity Analyses Would Help , 1985 .

[80]  Alison S. Tomlin,et al.  GUI-HDMR - A software tool for global sensitivity analysis of complex models , 2009, Environ. Model. Softw..

[81]  Kurt Maute,et al.  Level-set methods for structural topology optimization: a review , 2013 .

[82]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal , 1993 .

[83]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[84]  Mary C. Hill,et al.  Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models , 2014 .

[85]  Stefano Tarantola,et al.  Random balance designs for the estimation of first order global sensitivity indices , 2006, Reliab. Eng. Syst. Saf..

[86]  Paul G. Constantine,et al.  Active subspaces for sensitivity analysis and dimension reduction of an integrated hydrologic model , 2015, Comput. Geosci..

[87]  R. Spear.  Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis , 1980 .

[88]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..