Comparison of Surrogate-Based Uncertainty Quantification Methods for Computationally Expensive Simulators

Polynomial chaos and Gaussian process emulation are methods for surrogate-based uncertainty quantification, and have been developed independently in their respective communities over the last 25 years. Despite tackling similar problems in the field, to our knowledge there has yet to be a critical comparison of the two approaches in the literature. We begin by providing a detailed description of polynomial chaos and Gaussian process approaches for building a surrogate model of a black-box function. The accuracy of each surrogate method is then tested and compared for two simulators used in industry: a land-surface model (adJULES) and a launch vehicle controller (VEGACONTROL). We analyse surrogates built on experimental designs of various size and type to investigate their performance in a range of modelling scenarios. Specifically, polynomial chaos and Gaussian process surrogates are built on Sobol sequence and tensor grid designs. Their accuracy is measured by their ability to estimate the mean, standard deviation, exceedance probabilities and probability density function of the simulator output, as well as a root mean square error metric, based on an independent validation design. We find that one method does not unanimously outperform the other, but advantages can be gained in some cases, such that the preferred method depends on the modelling goals of the practitioner. Our conclusions are likely to depend somewhat on the modelling choices for the surrogates as well as the design strategy. We hope that this work will spark future comparisons of the two methods in their more advanced formulations and for different sampling strategies.

[1]  H. Najm,et al.  Uncertainty quantification in reacting-flow simulations through non-intrusive spectral projection , 2003 .

[2]  D. Xiu,et al.  Modeling uncertainty in flow simulations via generalized polynomial chaos , 2003 .

[3]  Jeroen A. S. Witteveen,et al.  Probabilistic collocation for period-1 limit cycle oscillations , 2008 .

[4]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[5]  M. Eldred,et al.  Evaluation of Non-Intrusive Approaches for Wiener-Askey Generalized Polynomial Chaos. , 2008 .

[6]  R. Ghanem,et al.  Stochastic Finite-Element Analysis of Seismic Soil-Structure Interaction , 2002 .

[7]  Jeremy E. Oakley,et al.  Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures , 2013, Technometrics.

[8]  A. O'Hagan,et al.  Polynomial Chaos : A Tutorial and Critique from a Statistician ’ s Perspective , 2013 .

[9]  R. Ghanem,et al.  A stochastic projection method for fluid flow. I: basic formulation , 2001 .

[10]  A. O'Hagan,et al.  Bayesian inference for the uncertainty distribution of computer model outputs , 2002 .

[11]  O. L. Maître,et al.  Uncertainty propagation in CFD using polynomial chaos decomposition , 2006 .

[12]  Jason L. Loeppky,et al.  Batch sequential designs for computer experiments , 2010 .

[13]  Dave Higdon,et al.  Combining Field Data and Computer Simulations for Calibration and Prediction , 2005, SIAM J. Sci. Comput..

[14]  Fabio Nobile,et al.  Stochastic Partial Differential Equations: Analysis and Computations a Multi Level Monte Carlo Method with Control Variate for Elliptic Pdes with Log- Normal Coefficients a Multi Level Monte Carlo Method with Control Variate for Elliptic Pdes with Log-normal Coefficients , 2022 .

[15]  Prathyush P. Menon,et al.  Cross-Entropy Based Probabilistic Analysis of VEGA Launcher Performance , 2015 .

[16]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[17]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[18]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[19]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[20]  Jangsun Baek,et al.  Efficient computation of maximum likelihood estimators in a spatial linear model with power exponential covariogram , 2001 .

[21]  P. Cox,et al.  The Joint UK Land Environment Simulator (JULES), model description – Part 2: Carbon fluxes and vegetation dynamics , 2011 .

[22]  Dongbin Xiu,et al.  High-Order Collocation Methods for Differential Equations with Random Inputs , 2005, SIAM J. Sci. Comput..

[23]  Daniel Busby,et al.  Hierarchical adaptive experimental design for Gaussian process emulators , 2009, Reliab. Eng. Syst. Saf..

[24]  Omar M. Knio,et al.  Sparse Pseudo Spectral Projection Methods with Directional Adaptation for Uncertainty Quantification , 2016, J. Sci. Comput..

[25]  Zhonghua Han,et al.  Efficient Uncertainty Quantification using Gradient-Enhanced Kriging , 2009 .

[26]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[27]  P. Cox,et al.  The Joint UK Land Environment Simulator (JULES), model description – Part 1: Energy and water fluxes , 2011 .

[28]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[29]  Fabio Nobile,et al.  A Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data , 2008, SIAM J. Numer. Anal..

[30]  M. Eldred,et al.  Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification , 2009 .

[31]  Fabio Nobile,et al.  Multi-index Monte Carlo: when sparsity meets sampling , 2014, Numerische Mathematik.

[32]  John C. Warner,et al.  Ocean forecasting in terrain-following coordinates: Formulation and skill assessment of the Regional Ocean Modeling System , 2008, J. Comput. Phys..

[33]  Sondipon Adhikari,et al.  Gaussian process emulators for the stochastic finite element method , 2011 .

[34]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[35]  G. Karniadakis,et al.  An adaptive multi-element generalized polynomial chaos method for stochastic differential equations , 2005 .

[36]  Michael Goldstein,et al.  History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble , 2013, Climate Dynamics.

[37]  George E. Karniadakis,et al.  Beyond Wiener–Askey Expansions: Handling Arbitrary PDFs , 2006, J. Sci. Comput..

[38]  T Watson Layne,et al.  Multidisciplinary Optimization of a Supersonic Transport Using Design of Experiments Theory and Response Surface Modeling , 1997 .

[39]  B. Sudret,et al.  An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis , 2010 .

[40]  H. Najm,et al.  A stochastic projection method for fluid flow II.: random process , 2002 .

[41]  Dongbin Xiu,et al.  Minimal multi-element stochastic collocation for uncertainty quantification of discontinuous functions , 2013, J. Comput. Phys..

[42]  N. Wiener.  The Homogeneous Chaos , 1938 .

[43]  Dongbin Xiu,et al.  A Stochastic Collocation Algorithm with Multifidelity Models , 2014, SIAM J. Sci. Comput..

[44]  C. Jones,et al.  The HadGEM2 family of Met Office Unified Model climate configurations , 2011 .

[45]  T. J. Mitchell,et al.  Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments , 1991 .

[46]  Loic Le Gratiet,et al.  Bayesian Analysis of Hierarchical Multifidelity Codes , 2011, SIAM/ASA J. Uncertain. Quantification.

[47]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[48]  Victor Picheny,et al.  A Nonstationary Space-Time Gaussian Process Model for Partially Converged Simulations , 2013, SIAM/ASA J. Uncertain. Quantification.

[49]  R. Ghanem,et al.  Stochastic Finite Elements: A Spectral Approach , 1990 .

[50]  H. Niederreiter.  Low-discrepancy and low-dispersion sequences , 1988 .

[51]  Dishi Liu,et al.  Quantification of Airfoil Geometry-Induced Aerodynamic Uncertainties - Comparison of Approaches , 2015, SIAM/ASA J. Uncertain. Quantification.

[52]  Jeremy E. Oakley,et al.  Diagnostics for Gaussian Process Emulators A full Bayesian approach , 2008 .

[53]  J. Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[54]  Peter S. Craig,et al.  Hierarchical Emulation: A Method for Modeling and Comparing Nested Simulators , 2016, SIAM/ASA J. Uncertain. Quantification.

[55]  V. Balabanov,et al.  Multidisciplinary optimisation of a supersonic transport using design of experiments theory and response surface modelling , 1997, The Aeronautical Journal (1968).

[56]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[57]  S. W. Kirkpatrick,et al.  Development and Validation of High Fidelity Vehicle Crash Simulation Models , 1999 .

[58]  Nathan M. Urban,et al.  A comparison of Latin hypercube and grid ensemble designs for the multivariate emulation of an Earth system model , 2010, Comput. Geosci..

[59]  A. O'Hagan,et al.  Probabilistic sensitivity analysis of complex models: a Bayesian approach , 2004 .

[60]  Bruno Sudret,et al.  Adaptive sparse polynomial chaos expansion based on least angle regression , 2011, J. Comput. Phys..

[61]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[62]  Habib N. Najm,et al.  Numerical Challenges in the Use of Polynomial Chaos Representations for Stochastic Processes , 2005, SIAM J. Sci. Comput..

[63]  Yves Deville,et al.  DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization , 2012 .

[64]  A. O'Hagan,et al.  Bayesian emulation of complex multi-output and dynamic computer models , 2010 .

[65]  Peter Challenor,et al.  Computational Statistics and Data Analysis the Effect of the Nugget on Gaussian Process Emulators of Computer Models , 2022 .

[66]  M. Lemaire,et al.  Stochastic finite element: a non intrusive approach by regression , 2006 .