An online Bayesian filtering framework for Gaussian process regression: Application to global surface temperature analysis

A novel Bayesian filtering for GP regression, compared to other GP variants.It reduces computation while improving accuracy for large data sets.GP-based state space model processes data efficiently in a sequential manner.Our online learning mechanism is a novel venue for parameter optimization in GP.An efficient and accurate expert system for global surface temperature analysis. Over the past centuries, global warming has gradually become one of the most significant issues in our life. Hence, it is crucial to analyze global surface temperature with an efficient and accurate model. Gaussian process (GP) is a popular nonparametric model, due to the power of Bayesian inference framework. However, the performance of GP is often deteriorated for large-scale data sets such as global surface temperature. In this work, we propose a novel online Bayesian filtering framework for large-scale GP regression. There are three contributions. Firstly, we develop a novel GP-based state space model to efficiently process data in a sequential manner. Secondly, based on our state space model, we design a marginalized particle filter to infer the latent function values and learn the model parameters online. It can efficiently reduce the computation burden of GP while improving the estimation accuracy in a recursive Bayesian inference framework. Finally, we successfully apply our approach to a number of synthetic data sets and the large-scale global surface temperature data set. The results show that our approach outperforms related GP variants, and it is an efficient and accurate expert system for global surface temperature analysis.

[1]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[2]  Yali Wang,et al.  Sequential Inference for Deep Gaussian Process , 2016, AISTATS.

[3]  Roger M. Goodall,et al.  Estimation of parameters in a linear state space model using a Rao-Blackwellised particle filter , 2004 .

[4]  Steven Reece,et al.  An introduction to Gaussian processes for the Kalman filter expert , 2010, 2010 13th International Conference on Information Fusion.

[5]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[6]  Carl E. Rasmussen,et al.  Understanding Probabilistic Sparse Gaussian Process Approximations , 2016, NIPS.

[7]  Miguel Lázaro-Gredilla,et al.  Kernel Recursive Least-Squares Tracker for Time-Varying Regression , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Marc Deisenroth.  Distributed Gaussian Processes for Large-Scale Probabilistic Regression , 2015 .

[9]  David Barber,et al.  Bayesian reasoning and machine learning , 2012 .

[10]  N. D. Freitas.  Rao-Blackwellised particle filtering for fault diagnosis , 2002 .

[11]  Arnaud Doucet,et al.  An overview of sequential Monte Carlo methods for parameter estimation in general state-space models , 2009 .

[12]  Carl E. Rasmussen,et al.  Sparse Spectrum Gaussian Process Regression , 2010, J. Mach. Learn. Res..

[13]  Kian Hsiang Low,et al.  Recent Advances in Scaling Up Gaussian Process Predictive Models for Large Spatiotemporal Data , 2014, DyDESS.

[14]  Sally Wood,et al.  Bayesian mixture of splines for spatially adaptive nonparametric regression , 2002 .

[15]  Mark J. Schervish,et al.  Nonstationary Covariance Functions for Gaussian Process Regression , 2003, NIPS.

[16]  Christian Plagemann,et al.  Gaussian processes for flexible robot learning , 2008 .

[17]  Carl E. Rasmussen,et al.  Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models , 2014, NIPS.

[18]  Arno Solin,et al.  State Space Methods for Efficient Inference in Student-t Process Regression , 2015, AISTATS.

[19]  Katherine L. Silversides,et al.  Fusing Gaussian Processes and Dynamic Time Warping for Improved Natural Gamma Signal Classification , 2016, Mathematical Geosciences.

[20]  Luc Van Gool,et al.  Learning Probabilistic Non-Linear Latent Variable Models for Tracking Complex Activities , 2011, NIPS.

[21]  Michael A. West,et al.  Combined Parameter and State Estimation in Simulation-Based Filtering , 2001, Sequential Monte Carlo Methods in Practice.

[22]  Hyejin Park,et al.  Parametric models and non-parametric machine learning models for predicting option prices: Empirical comparison study over KOSPI 200 Index options , 2014, Expert Syst. Appl..

[23]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[24]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[25]  Simon J. Godsill,et al.  An Overview of Existing Methods and Recent Advances in Sequential Monte Carlo , 2007, Proceedings of the IEEE.

[26]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[27]  Aki Vehtari,et al.  Gaussian process regression with Student-t likelihood , 2009, NIPS.

[28]  Thomas B. Schön,et al.  Marginalized particle filters for mixed linear/nonlinear state-space models , 2005, IEEE Transactions on Signal Processing.

[29]  Aki Vehtari,et al.  GPstuff: Bayesian modeling with Gaussian processes , 2013, J. Mach. Learn. Res..

[30]  Lehel Csató,et al.  Sparse On-Line Gaussian Processes , 2002, Neural Computation.

[31]  D. Mackay,et al.  Introduction to Gaussian processes , 1998 .

[32]  Neil D. Lawrence,et al.  Variationally Auto-Encoded Deep G aussian Processes , 2016, International Conference on Learning Representations.

[33]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[34]  Duy Nguyen-Tuong,et al.  Local Gaussian Process Regression for Real Time Online Model Learning , 2008, NIPS.

[35]  R. Kass,et al.  Bayesian curve-fitting with free-knot splines , 2001 .

[36]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[37]  M. Opper.  Sparse Online Gaussian Processes , 2008 .

[38]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[39]  Iain Murray,et al.  A framework for evaluating approximation methods for Gaussian process regression , 2012, J. Mach. Learn. Res..

[40]  Marc Peter Deisenroth,et al.  Distributed Gaussian Processes , 2015, ICML.

[41]  Neil J. Gordon,et al.  Editors: Sequential Monte Carlo Methods in Practice , 2001 .