Emulating dynamic non-linear simulators using Gaussian processes

Abstract The dynamic emulation of non-linear deterministic computer codes where the output is a time series, possibly multivariate, is examined. Such computer models simulate the evolution of some real-world phenomenon over time, for example models of the climate or the functioning of the human brain. The models we are interested in are highly non-linear and exhibit tipping points, bifurcations and chaotic behaviour. However, each simulation run could be too time-consuming to perform analyses that require many runs, including quantifying the variation in model output with respect to changes in the inputs. Therefore, Gaussian process emulators are used to approximate the output of the code. To do this, the flow map of the system under study is emulated over a short time period. Then, it is used in an iterative way to predict the whole time series. A number of ways are proposed to take into account the uncertainty of inputs to the emulators, after fixed initial conditions, and the correlation between them through the time series. The methodology is illustrated with two examples: the highly non-linear dynamical systems described by the Lorenz and van der Pol equations. In both cases, the predictive performance is relatively high and the measure of uncertainty provided by the method reflects the extent of predictability in each system.

[1]  Andrew M. Stuart,et al.  How Deep Are Deep Gaussian Processes? , 2017, J. Mach. Learn. Res..

[2]  A. O'Hagan,et al.  Bayesian emulation of complex multi-output and dynamic computer models , 2010 .

[3]  Henry P. Wynn,et al.  [Design and Analysis of Computer Experiments]: Rejoinder , 1989 .

[4]  SteinMichael.  Large sample properties of simulations using latin hypercube sampling , 1987 .

[5]  Luc Pronzato,et al.  Design of computer experiments: space filling and beyond , 2011, Statistics and Computing.

[6]  Geoffrey E. Hinton,et al.  Evaluation of Gaussian processes and other methods for non-linear regression , 1997 .

[7]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[8]  Neil D. Lawrence,et al.  Modelling transcriptional regulation using Gaussian Processes , 2006, NIPS.

[9]  Paul J. Birrell,et al.  Bayesian modeling to unmask and predict influenza A/H1N1pdm dynamics in London , 2011, Proceedings of the National Academy of Sciences.

[10]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[11]  Matthew Plumlee,et al.  Fast Prediction of Deterministic Functions Using Sparse Grid Experimental Designs , 2014, 1402.6350.

[12]  Coryn A. L. Bailer-Jones,et al.  PREDICTION OF DEFORMED AND ANNEALED MICROSTRUCTURES USING BAYESIAN NEURAL NETWORKS AND GAUSSIAN PROCESSES , 1997 .

[13]  E. Bruce Pitman,et al.  Computational Statistics and Data Analysis Mechanism-based Emulation of Dynamic Simulation Models: Concept and Application in Hydrology , 2022 .

[14]  Uwe D. Hanebeck,et al.  Analytic moment-based Gaussian process filtering , 2009, ICML '09.

[15]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[16]  A. O'Hagan,et al.  Gaussian process emulation of dynamic computer codes , 2009 .

[17]  Rachel T. Johnson,et al.  Design and analysis for the Gaussian process model , 2009, Qual. Reliab. Eng. Int..

[18]  Jeremy E. Oakley,et al.  Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures , 2013, Technometrics.

[19]  Yves Deville,et al.  DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization , 2012 .

[20]  G. Mann,et al.  Emulation of a complex global aerosol model to quantify sensitivity to uncertain parameters , 2011 .

[21]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[22]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[23]  J. O. Rawlings,et al.  Applied Regression Analysis: A Research Tool , 1988 .

[24]  B. Kendall.  Nonlinear Dynamics and Chaos , 2001 .

[25]  Malte Kuß,et al.  Gaussian process models for robust regression, classification, and reinforcement learning , 2006 .

[26]  Valentin Afraimovich,et al.  Origin and structure of the Lorenz attractor , 1977 .

[27]  David C. Woods,et al.  Emulation of Multivariate Simulators Using Thin-Plate Splines with Application to Atmospheric Dispersion , 2015, SIAM/ASA J. Uncertain. Quantification.

[28]  Agathe Girard,et al.  Propagation of uncertainty in Bayesian kernel models - application to multiple-step ahead forecasting , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[29]  Andy J. Keane,et al.  Recent advances in surrogate-based optimization , 2009 .

[30]  Robert Marsh,et al.  The probability of rapid climate change , 2004 .

[31]  D. Higdon,et al.  Computer Model Calibration Using High-Dimensional Output , 2008 .

[32]  A. OHagan,et al.  Bayesian analysis of computer code outputs: A tutorial , 2006, Reliab. Eng. Syst. Saf..

[33]  L. Petzold.  Automatic Selection of Methods for Solving Stiff and Nonstiff Systems of Ordinary Differential Equations , 1983 .

[34]  Peter G. Challenor,et al.  The probability of rapid climate change: II , 2004 .

[35]  Anton Schwaighofer,et al.  GPPS: A Gaussian Process Positioning System for Cellular Networks , 2003, NIPS.

[36]  Andreas Krause,et al.  Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies , 2008, J. Mach. Learn. Res..

[37]  Samik Raychaudhuri,et al.  Introduction to Monte Carlo simulation , 2008, 2008 Winter Simulation Conference.

[38]  Lea Fleischer,et al.  Design And Analysis Of Computer Experiments , 2016 .

[39]  M. J. Bayarri,et al.  Computer model validation with functional output , 2007, 0711.3271.

[40]  Ruichen Jin,et al.  On Sequential Sampling for Global Metamodeling in Engineering Design , 2002, DAC 2002.

[41]  John R. Terry,et al.  Classifying dynamic transitions in high dimensional neural mass models: A random forest approach , 2018, PLoS Comput. Biol..

[42]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[43]  Daniel B. Williamson,et al.  Evolving Bayesian Emulators for Structured Chaotic Time Series, with Application to Large Climate Models , 2014, SIAM/ASA J. Uncertain. Quantification.

[44]  Aditya Kamath,et al.  Neural networks vs Gaussian process regression for representing potential energy surfaces: A comparative study of fit quality and vibrational spectrum accuracy. , 2018, The Journal of chemical physics.

[45]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[46]  Idris A. Eckley,et al.  changepoint: An R Package for Changepoint Analysis , 2014 .

[47]  J. Rougier.  Efficient Emulators for Multivariate Deterministic Functions , 2008 .

[48]  R. F. Williams,et al.  The structure of Lorenz attractors , 1979 .

[49]  H. Stommel,et al.  Thermohaline Convection with Two Stable Regimes of Flow , 1961 .

[50]  Neil D. Lawrence,et al.  A Simple Approach to Ranking Differentially Expressed Gene Expression Time Courses through Gaussian Process Regression , 2011, BMC Bioinformatics.

[51]  Russell R. Barton,et al.  A review on design, modeling and applications of computer experiments , 2006 .

[52]  Iain Murray.  Introduction To Gaussian Processes , 2008 .

[53]  A. P. Dawid,et al.  Regression and Classification Using Gaussian Process Priors , 2009 .

[54]  Jakub Szymanik,et al.  Methods Results & Discussion , 2007 .

[55]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[56]  Jack P. C. Kleijnen,et al.  Kriging Metamodeling in Simulation: A Review , 2007, Eur. J. Oper. Res..

[57]  T. Simpson,et al.  Comparative studies of metamodeling techniques under multiple modeling criteria , 2000 .

[58]  D. Kavetski,et al.  Towards a Bayesian total error analysis of conceptual rainfall-runoff models: Characterising model error using storm-dependent parameters , 2006 .

[59]  Teuta Pilizota,et al.  Inferring time derivatives including cell growth rates using Gaussian processes , 2016, Nature Communications.

[60]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[61]  Karline Soetaert,et al.  Solving Differential Equations in R: Package deSolve , 2010 .

[62]  M. Stein.  Large sample properties of simulations using latin hypercube sampling , 1987 .

[63]  Michael A. West,et al.  A dynamic modelling strategy for Bayesian computer model emulation , 2009 .

[64]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.