Sensitivity analysis of environmental models: A systematic review with practical workflow

Sensitivity Analysis (SA) investigates how the variation in the output of a numerical model can be attributed to variations of its input factors. SA is increasingly being used in environmental modelling for a variety of purposes, including uncertainty assessment, model calibration and diagnostic evaluation, dominant control analysis and robust decision-making. In this paper we review the SA literature with the goal of providing: (i) a comprehensive view of SA approaches also in relation to other methodologies for model identification and application; (ii) a systematic classification of the most commonly used SA methods; (iii) practical guidelines for the application of SA. The paper aims at delivering an introduction to SA for non-specialist readers, as well as practical advice with best practice examples from the literature; and at stimulating the discussion within the community of SA developers and users regarding the setting of good practices and on defining priorities for future research. We present an overview of SA and its link to uncertainty analysis, model calibration and evaluation, robust decision-making.We provide a systematic review of existing approaches, which can support users in the choice of an SA method.We provide practical guidelines by developing a workflow for the application of SA and discuss critical choices.We give best practice examples from the literature and highlight trends and gaps for future research.

[1]  J. Doherty,et al.  Role of the calibration process in reducing model predictive error , 2005 .

[2]  Thorsten Wagener,et al.  Ten guidelines for effective data visualization in scientific publications , 2011, Environ. Model. Softw..

[3]  J. Kleijnen,et al.  Statistical analyses of scatterplots to identify important factors in large-scale simulations, 2: robustness of techniques , 1999 .

[4]  Andrea Saltelli,et al.  From screening to quantitative sensitivity analysis. A unified approach , 2011, Comput. Phys. Commun..

[5]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[6]  Florian Pappenberger,et al.  Multi-method global sensitivity analysis of flood inundation models. , 2008 .

[7]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[8]  P. Reed,et al.  Characterization of watershed model behavior across a hydroclimatic gradient , 2008 .

[9]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[10]  Wiley India.  Management Science: The Art Of Modeling With Spreadsheets , 2017 .

[11]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[12]  A. Saltelli,et al.  Sensitivity analysis of an environmental model: an application of different analysis methods , 1997 .

[13]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[14]  F. J. Davis,et al.  Illustration of Sampling‐Based Methods for Uncertainty and Sensitivity Analysis , 2002, Risk analysis : an official publication of the Society for Risk Analysis.

[15]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[16]  Paola Annoni,et al.  Estimation of global sensitivity indices for models with dependent variables , 2012, Comput. Phys. Commun..

[17]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[18]  S. Tarantola,et al.  Reduced‐complexity modeling of braided rivers: Assessing model performance by sensitivity analysis, calibration, and validation , 2013 .

[19]  Thorsten Wagener,et al.  Identifiability of transient storage model parameters along a mountain stream , 2013 .

[20]  Soroosh Sorooshian,et al.  The Analysis of Structural Identifiability: Theory and Application to Conceptual Rainfall-Runoff Models , 1985 .

[21]  Thorsten Wagener,et al.  A new approach to visualizing time-varying sensitivity indices for environmental model diagnostics across evaluation time-scales , 2014, Environ. Model. Softw..

[22]  Erwin Zehe,et al.  Inferring model structural deficits by analyzing temporal dynamics of model performance and parameter sensitivity , 2011 .

[23]  B. Croke,et al.  Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R , 2013 .

[24]  Cosimo Solidoro,et al.  Global sensitivity analysis of a shallow-water 3D eutrophication model , 1999 .

[25]  Alison L. Kay,et al.  Climate change and river flooding: Part 2 sensitivity characterisation for british catchments and example vulnerability assessments , 2013, Climatic Change.

[26]  Emanuele Borgonovo,et al.  A new uncertainty importance measure , 2007, Reliab. Eng. Syst. Saf..

[27]  Xiaobo Zhou,et al.  Global Sensitivity Analysis , 2017, Encyclopedia of GIS.

[28]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[29]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[30]  Patrick M. Reed,et al.  Advancing the identification and evaluation of distributed rainfall‐runoff models using global sensitivity analysis , 2007 .

[31]  Ronald A. Howard,et al.  Decision analysis: practice and promise , 1988 .

[32]  Agus Sudjianto,et al.  Relative entropy based method for global and regional sensitivity analysis in probabilistic design , 2004, DAC 2004.

[33]  Pekka Pyykkö,et al.  Introduction to the quantum chemistry 2012 issue. , 2012, Chemical reviews.

[34]  N. Fohrer,et al.  How to improve the representation of hydrological processes in SWAT for a lowland catchment – temporal analysis of parameter sensitivity and model performance , 2014 .

[35]  Harvey M. Wagner,et al.  Global Sensitivity Analysis , 1995, Oper. Res..

[36]  Jim W. Hall,et al.  Uncertainty-based sensitivity indices for imprecise probability distributions , 2006, Reliab. Eng. Syst. Saf..

[37]  Holger R. Maier,et al.  Relative magnitudes of sources of uncertainty in assessing climate change impacts on water supply security for the southern Adelaide water supply system , 2013 .

[38]  P. Reed,et al.  Hydrology and Earth System Sciences Discussions Comparing Sensitivity Analysis Methods to Advance Lumped Watershed Model Identification and Evaluation , 2022 .

[39]  Venkataramanan Balakrishnan,et al.  System identification: theory for the user (second edition): Lennart Ljung; Prentice-Hall, Englewood Cliffs, NJ, 1999, ISBN 0-13-656695-2 , 2002, Autom..

[40]  S. Grunwald,et al.  A global sensitivity analysis tool for the parameters of multivariable catchment models , 2006 .

[41]  Nong Shang,et al.  Parameter uncertainty and interaction in complex environmental models , 1994 .

[42]  Sergei S. Kucherenko,et al.  Derivative based global sensitivity measures and their link with global sensitivity indices , 2009, Math. Comput. Simul..

[43]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[44]  J. C. Helton,et al.  Statistical Analyses of Scatterplots to Identify Important Factors in Large-Scale Simulations, 1: Review and Comparison of Techniques , 1999 .

[45]  Peter N. Francis,et al.  Sensitivity analysis of dispersion modeling of volcanic ash from Eyjafjallajökull in May 2010 , 2012 .

[46]  Patrick M. Reed,et al.  A top-down framework for watershed model evaluation and selection under uncertainty , 2009, Environ. Model. Softw..

[47]  Mary C. Hill,et al.  Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models , 2014 .

[48]  R. Dawson,et al.  Sensitivity Analysis for Hydraulic Models , 2009 .

[49]  M. Morris,et al.  Transformations and invariance in the sensitivity analysis of computer experiments , 2014 .

[50]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[51]  S. Uhlenbrook,et al.  Sensitivity analyses of a distributed catchment model to verify the model structure , 2005 .

[52]  John Norton,et al.  An introduction to sensitivity assessment of simulation models , 2015, Environ. Model. Softw..

[53]  Stefano Tarantola,et al.  A General Probabilistic Framework for uncertainty and global sensitivity analysis of deterministic models: A hydrological case study , 2014, Environ. Model. Softw..

[54]  Soroosh Sorooshian,et al.  A framework for development and application of hydrological models , 2001, Hydrology and Earth System Sciences.

[55]  Stefano Tarantola,et al.  Contribution to the sample mean plot for graphical and numerical sensitivity analysis , 2009, Reliab. Eng. Syst. Saf..

[56]  Hoshin Vijai Gupta,et al.  A process‐based diagnostic approach to model evaluation: Application to the NWS distributed hydrologic model , 2008 .

[57]  David B. Stephenson,et al.  Statistical methods for interpreting Monte Carlo ensemble forecasts , 2000 .

[58]  Yuqiong Liu,et al.  Reconciling theory with observations: elements of a diagnostic approach to model evaluation , 2008 .

[59]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[60]  Susan Cuddy,et al.  Robust global sensitivity analysis of a river management model to assess nonlinear and interaction effects , 2014 .

[61]  Stefano Tarantola,et al.  Random balance designs for the estimation of first order global sensitivity indices , 2006, Reliab. Eng. Syst. Saf..

[62]  Willy Bauwens,et al.  Sobol' sensitivity analysis of a complex environmental model , 2011, Environ. Model. Softw..

[63]  Kandace J. Landreneau.  “ Sampling Strategies ” , 2005 .

[64]  Emanuele Borgonovo,et al.  Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? , 2014, Risk analysis : an official publication of the Society for Risk Analysis.

[65]  D. Higdon,et al.  Accelerating Markov Chain Monte Carlo Simulation by Differential Evolution with Self-Adaptive Randomized Subspace Sampling , 2009 .

[66]  K. Beven,et al.  Bayesian Estimation of Uncertainty in Runoff Prediction and the Value of Data: An Application of the GLUE Approach , 1996 .

[67]  Jeremy E. Oakley,et al.  A web-based tool for eliciting probability distributions from experts , 2014, Environ. Model. Softw..

[68]  Neil McIntyre,et al.  Towards reduced uncertainty in conceptual rainfall‐runoff modelling: dynamic identifiability analysis , 2003 .

[69]  Elizabeth B. Harper,et al.  Global sensitivity analysis for complex ecological models: a case study of riparian cottonwood population dynamics. , 2011, Ecological applications : a publication of the Ecological Society of America.

[70]  C. Tiedeman,et al.  Effective Groundwater Model Calibration: With Analysis of Data, Sensitivities, Predictions, and Uncertainty , 2007 .

[71]  Terry Andres.  Sampling methods and sensitivity analysis for large parameter sets , 1997 .

[72]  Paola Annoni,et al.  Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index , 2010, Comput. Phys. Commun..

[73]  R. Srinivasan,et al.  A global sensitivity analysis tool for the parameters of multi-variable catchment models , 2006 .

[74]  Robert J. Lempert,et al.  Comparing Algorithms for Scenario Discovery , 2008 .

[75]  R. Spear.  Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis , 1980 .

[76]  Emanuele Borgonovo,et al.  Sensitivity Analysis of Model Output with Input Constraints: A Generalized Rationale for Local Methods , 2008, Risk analysis : an official publication of the Society for Risk Analysis.

[77]  Soroosh Sorooshian,et al.  Response surface parameter sensitivity analysis methods for postcalibration studies , 1982 .

[78]  Laurent Menut,et al.  Applications of adjoint modelling in atmospheric chemistry: sensitivity and inverse modelling , 2000, Environ. Model. Softw..

[79]  Kwang-Il Ahn,et al.  A new approach for measuring uncertainty importance and distributional sensitivity in probabilistic safety assessment , 1994 .

[80]  W. James Shuttleworth,et al.  A fully multiple-criteria implementation of the Sobol' method for parameter sensitivity analysis , 2012 .

[81]  Agus Sudjianto,et al.  Relative Entropy Based Method for Probabilistic Sensitivity Analysis in Engineering Design , 2006 .

[82]  Jing Yang,et al.  Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis , 2011, Environ. Model. Softw..

[83]  A. Saltelli,et al.  Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques , 1990 .

[84]  Hoshin Vijai Gupta,et al.  Model identification for hydrological forecasting under uncertainty , 2005 .

[85]  Jing Wang,et al.  Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method , 2013, Environ. Model. Softw..

[86]  Patrick M. Reed,et al.  From maps to movies: High-resolution time-varying sensitivity analysis for spatially distributed watershed models , 2013 .

[87]  Andrea Saltelli,et al.  Sensitivity analysis didn't help. A practitioner's critique of the Stern review , 2010 .

[88]  Casey Brown,et al.  A Decision‐Analytic Approach to Managing Climate Risks: Application to the Upper Great Lakes 1 , 2011 .

[89]  Keith Beven,et al.  Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology , 2001 .

[90]  Brian J. Wagner,et al.  Experimental design for estimating parameters of rate‐limited mass transfer: Analysis of stream tracer studies , 1997 .

[91]  Joseph P. Romano,et al.  On the uniform asymptotic validity of subsampling and the bootstrap , 2012, 1204.2762.

[92]  Patrick Willems,et al.  Global sensitivity analysis of yield output from the water productivity model , 2014, Environ. Model. Softw..

[93]  Bart Nijssen,et al.  Monte Carlo sensitivity analysis of land surface parameters using the Variable Infiltration Capacity model , 2007 .

[94]  H. Christopher Frey,et al.  Recommended Practice Regarding Selection of Sensitivity Analysis Methods Applied to Microbial Food Safety Process Risk Models , 2005 .

[95]  K. Shuler,et al.  A STUDY OF THE SENSITIVITY OF COUPLED REACTION SYSTEMS TO UNCERTAINTIES IN RATE COEFFICIENTS . II , 2014 .

[96]  P. Reed,et al.  Sensitivity-guided reduction of parametric dimensionality for multi-objective calibration of watershed models , 2009 .

[97]  N. E. Dorsey.  The theory of the Earth inductor as an inclinometer , 1913 .

[98]  Thorsten Wagener,et al.  A vulnerability driven approach to identify adverse climate and land use change combinations for critical hydrologic indicator thresholds: Application to a watershed in Pennsylvania, USA , 2014 .

[99]  John P. Norton.  Algebraic sensitivity analysis of environmental models , 2008, Environ. Model. Softw..

[100]  Andrea Castelletti,et al.  Emulation techniques for the reduction and sensitivity analysis of complex environmental models , 2012, Environ. Model. Softw..

[101]  A. Saltelli,et al.  Update 1 of: Sensitivity analysis for chemical models. , 2012, Chemical reviews.

[102]  Emanuele Borgonovo,et al.  A Methodology for Determining Interactions in Probabilistic Safety Assessment Models by Varying One Parameter at a Time , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[103]  B. Iooss,et al.  Derivative based global sensitivity measures , 2014, 1412.2619.

[104]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[105]  Emanuele Borgonovo,et al.  Sampling strategies in density-based sensitivity analysis , 2012, Environ. Model. Softw..

[106]  A. Saltelli,et al.  Sensitivity analysis for chemical models. , 2005, Chemical reviews.

[107]  Jon C. Helton,et al.  Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal , 1993 .

[108]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[109]  S. Hora,et al.  A Robust Measure of Uncertainty Importance for Use in Fault Tree System Analysis , 1990 .

[110]  Florian Pappenberger,et al.  Multi‐method global sensitivity analysis (MMGSA) for modelling floodplain hydrological processes , 2008 .

[111]  ' RonaldL.Iman,et al.  An Investigation of Uncertainty and Sensitivity Analysis Techniques for Computer Models , 2006 .

[112]  D. Finegold.  Book and Resource Reviews , 2005 .

[113]  Francesca Pianosi,et al.  A simple and efficient method for global sensitivity analysis based on cumulative distribution functions , 2015, Environ. Model. Softw..

[114]  Keith Beven,et al.  Prophecy, reality and uncertainty in distributed hydrological modelling , 1993 .

[115]  R. Lempert,et al.  Shaping the Next One Hundred Years: New Methods for Quantitative Long-Term Policy Analysis , 2003 .

[116]  Suraje Dessai,et al.  Robust adaptation to climate change , 2010 .

[117]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[118]  Francesco Minunno,et al.  Selecting Parameters for Bayesian Calibration of a Process-Based Model: A Methodology Based on Canonical Correlation Analysis , 2013, SIAM/ASA J. Uncertain. Quantification.

[119]  William H. Press,et al.  Numerical recipes in C , 2002 .

[120]  P. Cox,et al.  Quantifying future climate change , 2012 .

[121]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[122]  Emanuele Borgonovo,et al.  Sensitivity analysis: A review of recent advances , 2016, Eur. J. Oper. Res..

[123]  J. Freer,et al.  Benchmarking observational uncertainties for hydrology: rainfall, river discharge and water quality , 2012 .

[124]  Jon C. Helton,et al.  Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models , 2009, Reliab. Eng. Syst. Saf..

[125]  E. Borgonovo.  Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches , 2006, Risk analysis : an official publication of the Society for Risk Analysis.

[126]  Patrick M. Reed,et al.  Identifying parametric controls and dependencies in integrated assessment models using global sensitivity analysis , 2014, Environ. Model. Softw..

[127]  T. G. Nguyen,et al.  Systematic testing of an integrated systems model for coastal zone management using sensitivity and uncertainty analyses , 2007, Environ. Model. Softw..

[128]  N. A. S. Hamm,et al.  Variance-based sensitivity analysis of the probability of hydrologically induced slope instability , 2006, Comput. Geosci..