Sparse On-Line Gaussian Processes

We develop an approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian on-line algorithm, together with a sequential construction of a relevant subsample of the data that fully specifies the prediction of the GP model. By using an appealing parameterization and projection techniques in a reproducing kernel Hilbert space, recursions for the effective parameters and a sparse gaussian approximation of the posterior process are obtained. This allows for both a propagation of predictions and Bayesian error measures. The significance and robustness of our approach are demonstrated on a variety of experiments.

[1]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[2]  G. Wahba.  Spline models for observational data , 1990 .

[3]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[4]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[5]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[6]  David Lowe,et al.  Tracking of non-stationary time-series using resource allocating RBF networks , 1996 .

[7]  Opper.  On-line versus Off-line Learning from Random Examples: General Results. , 1996, Physical review letters.

[8]  M. Gibbs,et al.  Efficient implementation of gaussian processes , 1997 .

[9]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[10]  L. Eon Bottou.  Online Learning and Stochastic Approximations , 1998 .

[11]  Manfred Opper,et al.  Finite-Dimensional Approximation of Gaussian Processes , 1998, NIPS.

[12]  David Barber,et al.  Bayesian Classification With Gaussian Processes , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  David Saad,et al.  On-Line Learning in Neural Networks , 1999 .

[14]  Léon Bottou,et al.  On-line learning and stochastic approximations , 1999 .

[15]  Hidemitsu Ogawa,et al.  RKHS-based functional analysis for exact incremental learning , 1999, Neurocomputing.

[16]  Matthias W. Seeger,et al.  Bayesian Model Selection for Support Vector Machines, Gaussian Processes and Other Kernel Classifiers , 1999, NIPS.

[17]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[18]  David Haussler,et al.  Probabilistic kernel regression models , 1999, AISTATS.

[19]  Michael E. Tipping.  The Relevance Vector Machine , 1999, NIPS.

[20]  Ole Winther,et al.  Efficient Approaches to Gaussian Process Classification , 1999, NIPS.

[21]  Noel A Cressie,et al.  Long-Lead Prediction of Pacific SSTs via Bayesian Dynamic Modeling , 2000 .

[22]  Volker Tresp,et al.  A Bayesian Committee Machine , 2000, Neural Computation.

[23]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[24]  Bernhard Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, International Conference on Machine Learning.

[25]  Christopher K. I. Williams,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[26]  Lemm,et al.  Bayesian approach to inverse quantum statistics , 2000, Physical review letters.

[27]  Dan Cornford,et al.  Structured neural network modelling of multi-valued functions for wind vector retrieval from satellite scatterometer measurements , 2000, Neurocomputing.

[28]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[29]  M. Opper,et al.  Gaussian processes and SVM: Mean field results and leave-one-out , 2000 .

[30]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[31]  Tong Zhang.  Approximation Bounds for Some Sparse Kernel Regression Algorithms , 2002, Neural Computation.

[32]  Simon Haykin,et al.  On Different Facets of Regularization Theory , 2002, Neural Computation.

[33]  Junbin Gao,et al.  SVM regression through variational methods and its sequential implementation , 2003, Neurocomputing.

[34]  Ole Winther,et al.  Tractable inference for probabilistic data models , 2003, Complex..

[35]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[36]  M. Opper,et al.  inverse problems: some new approaches , 2022 .