Modelling local and global phenomena with sparse Gaussian processes

Much recent work has concerned sparse approximations to speed up the Gaussian process regression from the unfavorable O(n) scaling in computational time to O(nm). Thus far, work has concentrated on models with one covariance function. However, in many practical situations additive models with multiple covariance functions may perform better, since the data may contain both long and short length-scale phenomena. The long length-scales can be captured with global sparse approximations, such as fully independent conditional (FIC), and the short length-scales can be modeled naturally by covariance functions with compact support (CS). CS covariance functions lead to naturally sparse covariance matrices, which are computationally cheaper to handle than full covariance matrices. In this paper, we propose a new sparse Gaussian process model with two additive components: FIC for the long length-scales and CS covariance function for the short length-scales. We give theoretical and experimental results and show that under certain conditions the proposed model has the same computational complexity as FIC. We also compare the model performance of the proposed model to additive models approximated by fully and partially independent conditional (PIC). We use real data sets and show that our model outperforms FIC and PIC approximations for data sets with two additive phenomena.

[1]  พงศ์ศักดิ์ บินสมประสงค์,et al.  FORMATION OF A SPARSE BUS IMPEDANCE MATRIX AND ITS APPLICATION TO SHORT CIRCUIT STUDY , 1980 .

[2]  H. Rue,et al.  Approximate Bayesian inference for hierarchical Gaussian Markov random field models , 2007 .

[3]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[4]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[5]  Zoubin Ghahramani,et al.  Local and global sparse Gaussian process approximations , 2007, AISTATS.

[6]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[7]  Edward Lloyd Snelson,et al.  Flexible and efficient Gaussian process models for machine learning , 2007 .

[8]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.

[9]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[10]  T. Gneiting Compactly Supported Correlation Functions , 2002 .

[11]  Christopher J Paciorek,et al.  Spatial modelling using a new class of nonstationary covariance functions , 2006, Environmetrics.

[12]  Amos Storkey,et al.  Efficient Covariance Matrix Methods for Bayesian Gaussian Processes and Hopfield Neural Networks , 1999 .

[13]  H. Niessner,et al.  On computing the inverse of a sparse matrix , 1983 .

[14]  Timothy A. Davis,et al.  Algorithm 837: AMD, an approximate minimum degree ordering algorithm , 2004, TOMS.

[15]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.