An Intercomparison of Sampling Methods for Uncertainty Quantification of Environmental Dynamic Models

Uncertainty quantification (UQ) of environmental dynamic models requires an efficient way to extract the information about the relationship between input parameter and model output. A uniformly scattered sample set is generally preferred over crude Monte Carlo sampling for its ability to explore the parameter space more effectively and efficiently. This paper compares eight commonly used uniform sampling methods along with the crude Monte Carlo sampling. The efficiency is measured by six uniformity metrics, while the effectiveness is measured by the goodness-of-fit of the surrogate models, and the sensitivity analysis and optimization results. We used two test problems: the Sobol’ g-function and the SAC-SMA hydrological model. The results show that among the sampling methods evaluated, the Good Lattice Points (GLP) and Symmetric Latin hypercube (SLH) have the highest uniformity scores, and the Ranked Gram-Schmidt (RGS) de-correlation algorithm can further improve the uniformity of the lattice sample sets. On the other hand, the Quasi-Monte-Carlo (QMC) methods, such as Halton and Sobol’ sequences, are not as uniform as their theoretical potential suggests when the number of sample points is low. Further, we found no clear relationship between the sampling methods used and their effectiveness, as the latter is affected by many factors other than the sampling methods, such as the choice of the surrogate modeling methods, sensitivity analysis and optimization methods, and the intrinsic properties of the dynamic models.

[1]  Donald E. Brown,et al.  Global Optimization With Multivariate Adaptive Regression Splines , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Stefano Tarantola,et al.  Evaluation of MARS modeling technique for sensitivity analysis of model output , 2010 .

[3]  Peter Winker,et al.  Centered L2-discrepancy of random sampling and Latin hypercube design, and construction of uniform designs , 2002, Math. Comput..

[4]  Boxin Tang Orthogonal Array-Based Latin Hypercubes , 1993 .

[5]  I. Sobol Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[6]  Runze Li,et al.  Design and Modeling for Computer Experiments , 2005 .

[7]  Yong Zhang,et al.  Uniform Design: Theory and Application , 2000, Technometrics.

[8]  Paul Bratley,et al.  Algorithm 659: Implementing Sobol's quasirandom sequence generator , 1988, TOMS.

[9]  H. Niederreiter Quasi-Monte Carlo methods and pseudo-random numbers , 1978 .

[10]  J HickernellF,et al.  The Uniform Design and Its Applications , 1995 .

[11]  有薗 育生,et al.  最適設備規模設定に関するDistribution Free Approach , 1999 .

[12]  Jon C. Helton,et al.  Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models , 2009, Reliab. Eng. Syst. Saf..

[13]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[14]  Wei Gong,et al.  Multi-objective parameter optimization of common land model using adaptive surrogate modeling , 2014 .

[15]  A. Owen Controlling correlations in latin hypercube samples , 1994 .

[16]  Indulata Shukla Application of Number Theory to Cryptography , 2014 .

[17]  Wei Gong,et al.  Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis , 2013 .

[18]  Jerome Sacks,et al.  Designs for Computer Experiments , 1989 .

[19]  Saltelli Andrea,et al.  Sensitivity Analysis for Nonlinear Mathematical Models. Numerical ExperienceSensitivity Analysis for Nonlinear Mathematical Models. Numerical Experience , 1995 .

[20]  Colas Schretter,et al.  Monte Carlo and Quasi-Monte Carlo Methods , 2016 .

[21]  R. Iman,et al.  A distribution-free approach to inducing rank correlation among input variables , 1982 .

[22]  Fred J. Hickernell,et al.  A generalized discrepancy and quadrature error bound , 1998, Math. Comput..

[23]  Sandra Milev,et al.  Assessing Parameter Importance in Decision Models. Application to Health Economic Evaluations , 2013 .

[24]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[25]  J. E. H. Shaw,et al.  A Quasirandom Approach to Integration in Bayesian Statistics , 1988 .

[26]  Samim Konjicija,et al.  Multi-objective parameter optimization of S-MAC protocol , 2014, 2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[27]  Soroosh Sorooshian,et al.  Constraining Land Surface and Atmospheric Parameters of a Locally Coupled Model Using Observational Data , 2005 .

[28]  H. Weyl Über die Gleichverteilung von Zahlen mod. Eins , 1916 .

[29]  Eva Riccomagno,et al.  Experimental Design and Observation for Large Systems , 1996, Journal of the Royal Statistical Society: Series B (Methodological).

[30]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[31]  Saltelli Andrea,et al.  Global Sensitivity Analysis: The Primer , 2008 .

[32]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[33]  K Ang,et al.  A NOTE ON UNIFORM DISTRIBUTION AND EXPERIMENTAL DESIGN , 1981 .

[34]  I. Sloan Lattice Methods for Multiple Integration , 1994 .

[35]  Wei Gong,et al.  An evaluation of adaptive surrogate modeling based optimization with two benchmark problems , 2014, Environ. Model. Softw..

[36]  R. Caflisch,et al.  Quasi-Monte Carlo integration , 1995 .

[37]  F. J. Hickernell Lattice rules: how well do they measure up? in random and quasi-random point sets , 1998 .

[38]  M. E. Johnson,et al.  Minimax and maximin distance designs , 1990 .

[39]  Henry P. Wynn,et al.  Maximum entropy sampling , 1987 .

[40]  S. Sorooshian,et al.  Effective and efficient algorithm for multiobjective optimization of hydrologic models , 2003 .

[41]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[42]  Yuan Wang,et al.  Some Applications of Number-Theoretic Methods in Statistics , 1994 .

[43]  K. Fang,et al.  Number-theoretic methods in statistics , 1993 .

[44]  Lauwerens Kuipers,et al.  Uniform distribution of sequences , 1974 .

[45]  I. Sobol On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[46]  Kenny Q. Ye,et al.  Algorithmic construction of optimal symmetric Latin hypercube designs , 2000 .

[47]  William Li,et al.  Columnwise-pairwise algorithms with applications to the construction of supersaturated designs , 1997 .