Distributed robust Gaussian Process regression

We study distributed and robust Gaussian Processes where robustness is introduced by a Gaussian Process prior on the function values combined with a Student-t likelihood. The posterior distribution is approximated by a Laplace Approximation, and together with concepts from Bayesian Committee Machines, we efficiently distribute the computations and render robust GPs on huge data sets feasible. We provide a detailed derivation and report on empirical results. Our findings on real and artificial data show that our approach outperforms existing baselines in the presence of outliers by using all available data.

[1]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[2]  Neil D. Lawrence,et al.  Variational inference for Student-t models: Robust Bayesian interpolation and generalised component analysis , 2005, Neurocomputing.

[3]  J. Geweke,et al.  Bayesian Treatment of the Independent Student- t Linear Model , 1993 .

[4]  T. Chai,et al.  Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature , 2014 .

[5]  C. Rasmussen,et al.  Approximations for Binary Gaussian Process Classification , 2008 .

[6]  Aki Vehtari,et al.  Gaussian process regression with Student-t likelihood , 2009, NIPS.

[7]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[8]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[9]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[10]  Volker Tresp,et al.  A Bayesian Committee Machine , 2000, Neural Computation.

[11]  Malte Kuß,et al.  Gaussian process models for robust regression, classification, and reinforcement learning , 2006 .

[12]  Carl E. Rasmussen,et al.  Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models , 2014, NIPS.

[13]  R. Baierlein.  Probability Theory: The Logic of Science , 2004 .

[14]  A. O'Hagan,et al.  On Outlier Rejection Phenomena in Bayes Inference , 1979 .

[15]  David J. Fleet,et al.  Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions , 2014, ArXiv.

[16]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[17]  Marc Peter Deisenroth,et al.  Distributed Gaussian Processes , 2015, ICML.

[18]  Sean B. Holden,et al.  Robust Regression with Twinned Gaussian Processes , 2007, NIPS.

[19]  James D. Annan,et al.  Geoscientific Model Development - a journal about models, for modellers , 2010 .

[20]  D. Rubinfeld,et al.  Hedonic housing prices and the demand for clean air , 1978 .

[21]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[22]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations , 2013, UAI.

[23]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[24]  Aki Vehtari,et al.  Robust Gaussian Process Regression with a Student-t Likelihood , 2011, J. Mach. Learn. Res..

[25]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[26]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[27]  Radford M. Neal.  Monte Carlo Implementation , 1996 .