Hierarchical Gaussian Process Models for Improved Metamodeling

Simulations are often used for the design of complex systems as they allow one to explore the design space without the need to build several prototypes. Over the years, the simulation accuracy, as well as the associated computational cost, has increased significantly, limiting the overall number of simulations during the design process. Therefore, metamodeling aims to approximate the simulation response with a cheap to evaluate mathematical approximation, learned from a limited set of simulator evaluations. Kernel-based methods using stationary kernels are nowadays widely used. In many problems, the smoothness of the function varies in space, which we call nonstationary behavior [20]. However, using stationary kernels for nonstationary responses can be inappropriate and result in poor models when combined with sequential design. We present the application of two recent techniques: Deep Gaussian Processes and Gaussian Processes with nonstationary kernel, which are better able to cope with these difficulties. We evaluate the method for nonstationary regression on a series of real-world problems, showing that these recent approaches outperform the standard Gaussian Processes with stationary kernels. Results show that these techniques are suitable for the simulation community, and we outline the variational inference method for the Gaussian Process with nonstationary kernel.

[1]  Tom Dhaene,et al.  Deep Gaussian Process metamodeling of sequentially sampled non-stationary response surfaces , 2017, 2017 Winter Simulation Conference (WSC).

[2]  Juho Rousu,et al.  Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo , 2015, AISTATS.

[3]  Peter Challenor,et al.  Computational Statistics and Data Analysis the Effect of the Nugget on Gaussian Process Emulators of Computer Models , 2022 .

[4]  Daniel Hernández-Lobato,et al.  Deep Gaussian Processes for Regression using Approximate Expectation Propagation , 2016, ICML.

[5]  Robert Lehmensiek,et al.  Adaptive sampling applied to multivariate, multiple output rational interpolation models with application to microwave circuits , 2002 .

[6]  Jack P. C. Kleijnen,et al.  Efficient global optimisation for black-box simulation via sequential intrinsic Kriging , 2018, J. Oper. Res. Soc..

[7]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[8]  Dirk Gorissen,et al.  Grid-enabled adaptive surrogate modeling for computer aided engineering , 2010 .

[9]  Barry L. Nelson,et al.  Stochastic kriging for simulation metamodeling , 2008, 2008 Winter Simulation Conference.

[10]  Alexis Boukouvalas,et al.  GPflow: A Gaussian Process Library using TensorFlow , 2016, J. Mach. Learn. Res..

[11]  Dirk Gorissen,et al.  A Novel Hybrid Sequential Design Strategy for Global Surrogate Modeling of Computer Experiments , 2011, SIAM J. Sci. Comput..

[12]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[13]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[14]  Jack P. C. Kleijnen,et al.  Stochastic Intrinsic Kriging for Simulation Metamodelling , 2015 .

[15]  LI X.RONG,et al.  Evaluation of estimation algorithms part I: incomprehensive measures of performance , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[16]  Jeong‐Soo Park.  Optimal Latin-hypercube designs for computer experiments , 1994 .

[17]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[18]  Tom Dhaene,et al.  A Fuzzy Hybrid Sequential Design Strategy for Global Surrogate Modeling of High-Dimensional Computer Experiments , 2015, SIAM J. Sci. Comput..

[19]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[20]  Lehel Csató,et al.  Sparse On-Line Gaussian Processes , 2002, Neural Computation.

[21]  Xi Chen,et al.  Enhancing Stochastic Kriging Metamodels with Gradient Estimators , 2013, Oper. Res..

[22]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[23]  Dirk Gorissen,et al.  Automatic model type selection with heterogeneous evolution: An application to RF circuit block modeling , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[24]  Dirk Gorissen,et al.  Sequential modeling of a low noise amplifier with neural networks and active learning , 2009, Neural Computing and Applications.

[25]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[26]  Adji B. Dieng,et al.  Variational Inference via χ Upper Bound Minimization , 2017 .

[27]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[28]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[29]  Andy J. Keane,et al.  Engineering Design via Surrogate Modelling - A Practical Guide , 2008 .

[30]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[31]  Tirthankar Dasgupta,et al.  Sequential Exploration of Complex Surfaces Using Minimum Energy Designs , 2015, Technometrics.

[32]  Daniel J. Fonseca,et al.  Simulation metamodeling through artificial neural networks , 2003 .

[33]  Neil D. Lawrence,et al.  Nested Variational Compression in Deep Gaussian Processes , 2014, 1412.1370.

[34]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[35]  Samuel Kaski,et al.  Non-Stationary Spectral Kernels , 2017, NIPS.

[36]  Eric Petit,et al.  Adaptive sampling for performance characterization of application kernels , 2013, Concurr. Comput. Pract. Exp..

[37]  Dong-Hoon Choi,et al.  Kriging interpolation methods in geostatistics and DACE model , 2002 .

[38]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[39]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[40]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[41]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[42]  Ulrike von Luxburg,et al.  Lens Depth Function and k-Relative Neighborhood Graph: Versatile Tools for Ordinal Data Analysis , 2016, J. Mach. Learn. Res..