Sequential process convolution Gaussian process models via particle learning

The process convolution framework for constructing a Gaussian Process (GP) model is a computationally efficient approach for larger datasets in lower dimensions. Bayesian inference or specifically, Markov chain Monte Carlo, is commonly used for estimating the parameters of this model. However, applications where data arrive sequentially require re-running the Markov chain for each new data arrival, which can be computationally inefficient. This paper presents a sequential inference method for the process convolution GP model based on a Sequential Monte Carlo method called Particle Learning. This model is illustrated on a synthetic example and an optimization problem in hydrology.

[1]  B. Sansó,et al.  A Spatio-Temporal Model for Mean, Anomaly, and Trend Fields of North Atlantic Sea Surface Temperature , 2009 .

[2]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[3]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[4]  Sw. Banerjee,et al.  Hierarchical Modeling and Analysis for Spatial Data , 2003 .

[5]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[6]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[7]  David A. Cohn,et al.  Training Connectionist Networks with Queries and Selective Sampling , 1989, NIPS.

[8]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[9]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[10]  M. Pitt,et al.  Filtering via Simulation: Auxiliary Particle Filters , 1999 .

[11]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[12]  Thomas J. Santner,et al.  Design and analysis of computer experiments , 1998 .

[13]  Stephan R. Sain,et al.  spam: A Sparse Matrix R Package with Emphasis on MCMC Methods for Gaussian Markov Random Fields , 2010 .

[14]  Robert B. Gramacy,et al.  Adaptive Design and Analysis of Supercomputer Experiments , 2008, Technometrics.

[15]  Robert B. Gramacy,et al.  Dynamic Trees for Learning and Design , 2009, 0912.1586.

[16]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[17]  Stanley H. Cohen,et al.  Design and Analysis , 2010 .

[18]  Robert B. Gramacy,et al.  Cases for the nugget in modeling computer experiments , 2010, Statistics and Computing.

[19]  Klaus Obermayer,et al.  Gaussian Process Regression: Active Data Selection and Test Point Rejection , 2000, DAGM-Symposium.

[20]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[21]  G. Storvik.  Particle filters in state space models with the presence of unknown static parameters YYYY No org found YYY , 2000 .

[22]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[23]  M. West,et al.  Bayesian forecasting and dynamic models , 1989 .

[24]  Wolfram Burgard,et al.  Efficient Failure Detection on Mobile Robots Using Particle Filters with Gaussian Process Proposals , 2007, IJCAI.

[25]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[26]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[27]  Nicholas G. Polson,et al.  Particle Learning and Smoothing , 2010, 1011.1098.

[28]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[29]  A. P. Dawid,et al.  Regression and Classification Using Gaussian Process Priors , 2009 .

[30]  Michael A. West,et al.  Combined Parameter and State Estimation in Simulation-Based Filtering , 2001, Sequential Monte Carlo Methods in Practice.

[31]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.

[32]  Herbert K. H. Lee,et al.  Bayesian Guided Pattern Search for Robust Local Optimization , 2009, Technometrics.

[33]  Herbert K. H. Lee,et al.  Multiscale Modeling: A Bayesian Perspective , 2007 .

[34]  Eli Shamir,et al.  Learning Using Query by Committee, Linear Separation and Random Walks , 2002 .

[35]  Waley Wei Jie Liang,et al.  Bayesian nonstationary Gaussian process models via treed process convolutions , 2018, Advances in Data Analysis and Classification.

[36]  Robert B. Gramacy,et al.  tgp: An R Package for Bayesian Nonstationary, Semiparametric Nonlinear Regression and Design by Treed Gaussian Process Models , 2007 .

[37]  Dieter Fox,et al.  GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models , 2008, IROS.

[38]  Robert B. Gramacy,et al.  Optimization Under Unknown Constraints , 2010, 1004.4027.

[39]  D. Higdon,et al.  Exploring Space-Time Structure in Ozone Concentration Using a Dynamic Process Convolution Model , 2002 .

[40]  D. Higdon.  Space and Space-Time Modeling using Process Convolutions , 2002 .

[41]  Robert B. Gramacy,et al.  Particle Learning of Gaussian Process Models for Sequential Design and Optimization , 2009, 0909.5262.

[42]  N. Chopin.  A sequential particle filter method for static models , 2002 .

[43]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[44]  David Higdon,et al.  A Primer on Space-Time Modeling from a Bayesian Perspective , 2006 .

[45]  Christopher J Paciorek,et al.  Spatial modelling using a new class of nonstationary covariance functions , 2006, Environmetrics.

[46]  L. Shawn Matott,et al.  Application of MATLAB and Python optimizers to two case studies involving groundwater flow and contaminant transport modeling , 2011, Comput. Geosci..

[47]  David Higdon,et al.  A process-convolution approach to modelling temperatures in the North Atlantic Ocean , 1998, Environmental and Ecological Statistics.

[48]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.