Global sensitivity analysis for high-dimensional problems: How to objectively group factors and measure robustness and convergence while reducing computational cost

Abstract Dynamical earth and environmental systems models are typically computationally intensive and highly parameterized with many uncertain parameters. Together, these characteristics severely limit the applicability of Global Sensitivity Analysis (GSA) to high-dimensional models because very large numbers of model runs are typically required to achieve convergence and provide a robust assessment. Paradoxically, only 30 percent of GSA applications in the environmental modelling literature have investigated models with more than 20 parameters, suggesting that GSA is under-utilized on problems for which it should prove most useful. We develop a novel grouping strategy, based on bootstrap-based clustering, that enables efficient application of GSA to high-dimensional models. We also provide a new measure of robustness that assesses GSA stability and convergence. For two models, having 50 and 111 parameters, we show that grouping-enabled GSA provides results that are highly robust to sampling variability, while converging with a much smaller number of model runs.

[1]  Huaicheng Guo,et al.  Global sensitivity analysis of a three-dimensional nutrients-algae dynamic model for a large shallow lake , 2016 .

[2]  Joseph H. A. Guillaume,et al.  Characterising performance of environmental models , 2013, Environ. Model. Softw..

[3]  David E. Irwin,et al.  Finding a "Kneedle" in a Haystack: Detecting Knee Points in System Behavior , 2011, 2011 31st International Conference on Distributed Computing Systems Workshops.

[4]  Mats Nilsson,et al.  Parameter interactions and sensitivity analysis for modelling carbon heat and water fluxes in a natural peatland, using CoupModel v5 , 2016 .

[5]  Soroosh Sorooshian,et al.  Exploring parameter sensitivities of the land surface using a locally coupled land-atmosphere model , 2004 .

[6]  Fabrice Gamboa,et al.  Sensitivity analysis for multidimensional and functional outputs , 2013, 1311.1797.

[7]  Edoardo Patelli,et al.  Global sensitivity of structural variability by random sampling , 2010, Comput. Phys. Commun..

[8]  J. H. Ward Hierarchical Grouping to Optimize an Objective Function , 1963 .

[9]  Saman Razavi,et al.  Multicriteria sensitivity analysis as a diagnostic tool for understanding model behaviour and characterizing model uncertainty , 2017 .

[10]  Andrea Saltelli,et al.  An effective screening design for sensitivity analysis of large models , 2007, Environ. Model. Softw..

[11]  G. W. Milligan,et al.  A study of standardization of variables in cluster analysis , 1988 .

[12]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[13]  Alain Pietroniro,et al.  Development of the MESH modelling system for hydrological ensemble forecasting of the Laurentian Great Lakes at the regional scale , 2006 .

[14]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[15]  Emanuele Borgonovo,et al.  Model emulation and moment-independent sensitivity analysis: An application to environmental modelling , 2012, Environ. Model. Softw..

[16]  Jing Wang,et al.  Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method , 2013, Environ. Model. Softw..

[17]  Ingmar Nopens,et al.  Assessing the convergence of LHS Monte Carlo simulations of wastewater treatment models. , 2011, Water science and technology : a journal of the International Association on Water Pollution Research.

[18]  Saman Razavi,et al.  Progressive Latin Hypercube Sampling: An efficient approach for robust sampling-based analysis of environmental models , 2017, Environ. Model. Softw..

[19]  Ming Ye,et al.  Global sensitivity analysis in hydrological modeling: Review of concepts, methods, theoretical framework, and applications , 2015 .

[20]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[21]  I. Sobol,et al.  Sensitivity Measures, ANOVA-like Techniques and the Use of Bootstrap , 1997 .

[22]  R. Sakia The Box-Cox transformation technique: a review , 1992 .

[23]  Marc B. Neumann,et al.  Global sensitivity analysis in wastewater applications: A comprehensive comparison of different methods , 2013, Environ. Model. Softw..

[24]  Paola Annoni,et al.  Sixth International Conference on Sensitivity Analysis of Model Output How to avoid a perfunctory sensitivity analysis , 2010 .

[25]  Yoshikazu Terada,et al.  Clustering for high-dimension, low-sample size data using distance vectors , 2013, ArXiv.

[26]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory , 2016 .

[27]  Yuqiong Liu,et al.  Reconciling theory with observations: elements of a diagnostic approach to model evaluation , 2008 .

[28]  William Becker,et al.  Applications of Dynamic Trees to Sensitivity Analysis , 2015 .

[29]  Biagio Ciuffo,et al.  A Sensitivity-Analysis-Based Approach for the Calibration of Traffic Simulation Models , 2014, IEEE Transactions on Intelligent Transportation Systems.

[30]  Trupti M. Kodinariya,et al.  Review on determining number of Cluster in K-Means Clustering , 2013 .

[31]  Francesca Pianosi,et al.  Global Sensitivity Analysis of environmental models: Convergence and validation , 2016, Environ. Model. Softw..

[32]  Emanuele Borgonovo,et al.  A new uncertainty importance measure , 2007, Reliab. Eng. Syst. Saf..

[33]  Zhongbo Yu,et al.  Parameter sensitivity analysis and optimization of Noah land surface model with field measurements from Huaihe River Basin, China , 2015, Stochastic Environmental Research and Risk Assessment.

[34]  Holger R. Maier,et al.  An inverse approach to perturb historical rainfall data for scenario-neutral climate impact studies , 2016 .

[35]  Juliane Mai,et al.  Use of eigendecomposition in a parameter sensitivity analysis of the Community Land Model , 2013 .

[36]  Bryan A. Tolson,et al.  Review of surrogate modeling in water resources , 2012 .

[37]  Alain Pietroniro,et al.  Grouped Response Units for Distributed Hydrologic Modeling , 1993 .

[38]  G. A. Young,et al.  Recent Developments in Bootstrap Methodology , 2003 .

[39]  Antonio Marcomini,et al.  Modelling ecological and human exposure to POPs in Venice lagoon - Part II: Quantitative uncertainty and sensitivity analysis in coupled exposure models. , 2016, The Science of the total environment.

[40]  G. N. Lance,et al.  A General Theory of Classificatory Sorting Strategies: 1. Hierarchical Systems , 1967, Comput. J..

[41]  Saman Razavi,et al.  Enhanced identification of a hydrologic model using streamflow and satellite water storage data: A multicriteria sensitivity analysis and optimization approach , 2017 .

[42]  W. James Shuttleworth,et al.  A fully multiple-criteria implementation of the Sobol' method for parameter sensitivity analysis , 2012 .

[43]  Jing Yang,et al.  Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis , 2011, Environ. Model. Softw..

[44]  Patrick M. Reed,et al.  Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models , 2013 .

[45]  Patrick Willems,et al.  Global sensitivity analysis of yield output from the water productivity model , 2014, Environ. Model. Softw..

[46]  P. Sneath The application of computers to taxonomy. , 1957, Journal of general microbiology.

[47]  Sergei S. Kucherenko,et al.  Derivative based global sensitivity measures and their link with global sensitivity indices , 2009, Math. Comput. Simul..

[48]  Vipin Kumar,et al.  Introduction to Data Mining , 2022, Data Mining and Machine Learning Applications.

[49]  Laura Ferreira,et al.  A Comparison of Hierarchical Methods for Clustering Functional Data , 2009, Commun. Stat. Simul. Comput..

[50]  Casey Brown,et al.  Sustainable water management under future uncertainty with eco-engineering decision scaling , 2016 .

[51]  Wei Gong,et al.  Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis , 2013 .

[52]  Patrick M. Reed,et al.  Advancing the identification and evaluation of distributed rainfall‐runoff models using global sensitivity analysis , 2007 .

[53]  Tal Galili,et al.  dendextend: an R package for visualizing, adjusting and comparing trees of hierarchical clustering , 2015, Bioinform..

[54]  Marc B. Neumann,et al.  Global sensitivity analysis for urban water quality modelling: terminology, convergence and comparison of different methods. , 2015 .

[55]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[56]  Marten Scheffer,et al.  Beyond bifurcation: using complex models to understand and predict abrupt climate change , 2016 .

[57]  Jeffrey C. Lagarias,et al.  Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions , 1998, SIAM J. Optim..

[58]  F. Baker Stability of Two Hierarchical Grouping Techniques Case I: Sensitivity to Data Errors , 1974 .

[59]  R. Daniel Meyer,et al.  An Analysis for Unreplicated Fractional Factorials , 1986 .

[60]  Linda Trocine,et al.  An overview of newer, advanced screening methods for the initial phase in an experimental design , 2001, Proceeding of the 2001 Winter Simulation Conference (Cat. No.01CH37304).

[61]  B. Efron Bootstrap Methods: Another Look at the Jackknife , 1979 .

[62]  Omar Ali Eweys,et al.  Integrating WOFOST and Noah LSM for modeling maize production and soil moisture with sensitivity analysis, in the east of The Netherlands , 2017 .

[63]  Saman Razavi,et al.  What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models , 2015 .

[64]  C. Fortuin,et al.  Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory , 1973 .

[65]  A. Saltelli,et al.  A quantitative model-independent method for global sensitivity analysis of model output , 1999 .

[66]  Olivier Klepper,et al.  Multivariate aspects of model uncertainty analysis: tools for sensitivity analysis and calibration , 1997 .

[67]  J. Hair Multivariate data analysis , 1972 .

[68]  Terry Andres Sampling methods and sensitivity analysis for large parameter sets , 1997 .

[69]  Wei Gong,et al.  An Intercomparison of Sampling Methods for Uncertainty Quantification of Environmental Dynamic Models , 2015 .

[70]  Andrew J. Wade,et al.  Does increased hydrochemical model complexity decrease robustness , 2012 .

[71]  Soroosh Sorooshian,et al.  Sensitivity analysis of a land surface scheme using multicriteria methods , 1999 .

[72]  D. Kirschner,et al.  A methodology for performing global uncertainty and sensitivity analysis in systems biology. , 2008, Journal of theoretical biology.

[73]  Stefano Tarantola,et al.  Sensitivity analysis practices: Strategies for model-based inference , 2006, Reliab. Eng. Syst. Saf..

[74]  Joseph H. A. Guillaume,et al.  An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together? , 2016, Environ. Model. Softw..

[75]  B. Everitt,et al.  A Monte Carlo Study of the Recovery of Cluster Structure in Binary Data by Hierarchical Clustering Techniques. , 1987, Multivariate behavioral research.

[76]  Robert R. Sokal,et al.  A statistical method for evaluating systematic relationships , 1958 .

[77]  Stefano Tarantola,et al.  Random balance designs for the estimation of first order global sensitivity indices , 2006, Reliab. Eng. Syst. Saf..

[78]  Willy Bauwens,et al.  Sobol' sensitivity analysis of a complex environmental model , 2011, Environ. Model. Softw..

[79]  David Makowski,et al.  Multivariate sensitivity analysis to measure global contribution of input factors in dynamic models , 2011, Reliab. Eng. Syst. Saf..

[80]  H. Gupta,et al.  A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application , 2016 .

[81]  S. C. Johnson Hierarchical clustering schemes , 1967, Psychometrika.

[82]  T. Sørensen,et al.  A method of establishing group of equal amplitude in plant sociobiology based on similarity of species content and its application to analyses of the vegetation on Danish commons , 1948 .

[83]  I. Sobol,et al.  About the use of rank transformation in sensitivity analysis of model output , 1995 .

[84]  Soroosh Sorooshian,et al.  Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information , 1998 .

[85]  Francesca Pianosi,et al.  A simple and efficient method for global sensitivity analysis based on cumulative distribution functions , 2015, Environ. Model. Softw..

[86]  Saman Razavi,et al.  Insights into sensitivity analysis of Earth and environmental systems models: On the impact of parameter perturbation scale , 2017, Environ. Model. Softw..

[87]  Francesca Pianosi,et al.  Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model , 2017, Environ. Model. Softw..

[88]  Saman Razavi,et al.  Improved Understanding of River Ice Processes Using Global Sensitivity Analysis Approaches , 2017 .

[89]  R. Sokal,et al.  THE COMPARISON OF DENDROGRAMS BY OBJECTIVE METHODS , 1962 .