Practical Experience of Sensitivity Analysis: Comparing Six Methods, on Three Hydrological Models, with Three Performance Criteria

Currently, practically no modeling study is expected to be carried out without some form of Sensitivity Analysis (SA). At the same time, there is a large number of various methods and it is not always easy for practitioners to choose one. The aim of this paper is to briefly review main classes of SA methods, and to present the results of the practical comparative analysis of applying them. Six different global SA methods: Sobol, eFAST (extended Fourier Amplitude Sensitivity Test), Morris, LH-OAT, RSA (Regionalized Sensitivity Analysis), and PAWN are tested on three conceptual rainfall-runoff models with varying complexity: (GR4J, Hymod, and HBV) applied to the case study of Bagmati basin (Nepal). The methods are compared with respect to effectiveness, efficiency, and convergence. A practical framework of selecting and using the SA methods is presented. The result shows that, first of all, all the six SA methods are effective. Morris and LH-OAT methods are the most efficient methods in computing SI and ranking. eFAST performs better than Sobol, and thus it can be seen as its viable alternative for Sobol. PAWN and RSA methods have issues of instability, which we think are due to the ways Cumulative Distribution Functions (CDFs) are built, and using Kolmogorov–Smirnov statistics to compute Sensitivity Indices. All the methods require sufficient number of runs to reach convergence. Difference in efficiency of different methods is an inevitable consequence of the differences in the underlying principles. For SA of hydrological models, it is recommended to apply the presented practical framework assuming the use of several methods, and to explicitly take into account the constraints of effectiveness, efficiency (including convergence), ease of use, and availability of software.

[1]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[2]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[3]  William Stafford Noble,et al.  Support vector machine , 2013 .

[4]  K. Shuler,et al.  A STUDY OF THE SENSITIVITY OF COUPLED REACTION SYSTEMS TO UNCERTAINTIES IN RATE COEFFICIENTS . II , 2014 .

[5]  A. Saltelli,et al.  An alternative way to compute Fourier amplitude sensitivity test (FAST) , 1998 .

[6]  G. Hornberger,et al.  Approach to the preliminary analysis of environmental systems , 1981 .

[7]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[8]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[9]  R. C. Spear,et al.  The application of Kolmogorov–Rényi statistics to problems of parameter uncertainty in systems design† , 1970 .

[10]  Peter C. Young,et al.  Water quality in river systems: Monte‐Carlo Analysis , 1979 .

[11]  Pebesma Edzer,et al.  Managing uncertainty in integrated environmental modelling frameworks: The UncertWeb framework , 2012 .

[12]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory , 2016 .

[13]  L. S. Pereira,et al.  Crop evapotranspiration : guidelines for computing crop water requirements , 1998 .

[14]  Emanuele Borgonovo,et al.  Global sensitivity measures from given data , 2013, Eur. J. Oper. Res..

[15]  F. Massey The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .

[16]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[17]  Francesca Pianosi,et al.  A simple and efficient method for global sensitivity analysis based on cumulative distribution functions , 2015, Environ. Model. Softw..

[18]  Kwang-Il Ahn,et al.  A new approach for measuring uncertainty importance and distributional sensitivity in probabilistic safety assessment , 1994 .

[19]  Charles Tong,et al.  Refinement strategies for stratified sampling methods , 2006, Reliab. Eng. Syst. Saf..

[20]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application , 2016 .

[21]  C. Lemieux Monte Carlo and Quasi-Monte Carlo Sampling , 2009 .

[22]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[23]  R. Spear Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis , 1980 .

[24]  Patrick M. Reed,et al.  Advancing the identification and evaluation of distributed rainfall‐runoff models using global sensitivity analysis , 2007 .

[25]  Soroosh Sorooshian,et al.  A framework for development and application of hydrological models , 2001, Hydrology and Earth System Sciences.

[26]  Dimitri Solomatine,et al.  A novel method to estimate model uncertainty using machine learning techniques , 2009 .

[27]  Agus Sudjianto,et al.  Relative Entropy Based Method for Probabilistic Sensitivity Analysis in Engineering Design , 2006 .

[28]  Francesca Pianosi,et al.  Global Sensitivity Analysis of environmental models: Convergence and validation , 2016, Environ. Model. Softw..

[29]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[30]  Jing Yang,et al.  Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis , 2011, Environ. Model. Softw..

[31]  Anthony J. Jakeman,et al.  CALIBRATION AND RELIABILITY OF AN AQUIFER SYSTEM MODEL USING GENERALIZED SENSITIVITY ANALYSIS , 1990 .

[32]  C. Perrin,et al.  Improvement of a parsimonious model for streamflow simulation , 2003 .

[33]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[34]  F. Pappenberger,et al.  Ignorance is bliss: Or seven reasons not to use uncertainty analysis , 2006 .

[35]  Göran Lindström,et al.  Development and test of the distributed HBV-96 hydrological model , 1997 .

[36]  Jim W. Hall,et al.  Sensitivity analysis of environmental models: A systematic review with practical workflow , 2014, Environ. Model. Softw..

[37]  R. Srinivasan,et al.  A global sensitivity analysis tool for the parameters of multi-variable catchment models , 2006 .

[38]  Dan Cornford,et al.  Managing uncertainty in integrated environmental modelling: The UncertWeb framework , 2013, Environ. Model. Softw..

[39]  Saman Razavi,et al.  What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models , 2015 .

[40]  Ming Ye,et al.  Global sensitivity analysis in hydrological modeling: Review of concepts, methods, theoretical framework, and applications , 2015 .

[41]  Stefano Tarantola,et al.  A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output , 1999, Technometrics.

[42]  Florian Pappenberger,et al.  Multi-method global sensitivity analysis of flood inundation models. , 2008 .

[43]  Emanuele Borgonovo,et al.  A new uncertainty importance measure , 2007, Reliab. Eng. Syst. Saf..

[44]  Wei Chu,et al.  A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model , 2014, Environ. Model. Softw..

[45]  D. Cacuci,et al.  A Comparative Review of Sensitivity and Uncertainty Analysis of Large-Scale Systems—II: Statistical Methods , 2004 .

[46]  Stefano Tarantola,et al.  A General Probabilistic Framework for uncertainty and global sensitivity analysis of deterministic models: A hydrological case study , 2014, Environ. Model. Softw..

[47]  Francesca Pianosi,et al.  A Matlab toolbox for Global Sensitivity Analysis , 2015, Environ. Model. Softw..