On Approximate Inference for Generalized Gaussian Process Models

A generalized Gaussian process model (GGPM) is a unifying framework that encompasses many existing Gaussian process (GP) models, such as GP regression, classification, and counting. In the GGPM framework, the observation likelihood of the GP model is itself parameterized using the exponential family distribution (EFD). In this paper, we consider efficient algorithms for approximate inference on GGPMs using the general form of the EFD. A particular GP model and its associated inference algorithms can then be formed by changing the parameters of the EFD, thus greatly simplifying its creation for task-specific output domains. We demonstrate the efficacy of this framework by creating several new GP models for regressing to non-negative reals and to real intervals. We also consider a closed-form Taylor approximation for efficient inference on GGPMs, and elaborate on its connections with other model-specific heuristic closed-form approximations. Finally, we present a comprehensive set of experiments to compare approximate inference algorithms on a wide variety of GGPMs.

[1]  Marina Vannucci,et al.  Variable Selection for Nonparametric Gaussian Process Priors: Models and Computational Strategies. , 2011, Statistical science : a review journal of the Institute of Mathematical Statistics.

[2]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[3]  Reiner Lenz,et al.  Evaluation and unification of some methods for estimating reflectance spectra from RGB images. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[4]  Carl E. Rasmussen,et al.  Learning Depth from Stereo , 2004, DAGM-Symposium.

[5]  T. Minka,et al.  A useful distribution for fitting discrete data: revival of the Conway–Maxwell–Poisson distribution , 2005 .

[6]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[7]  R. Christensen,et al.  A New Perspective on Priors for Generalized Linear Models , 1996 .

[8]  Trevor Darrell,et al.  Sparse probabilistic regression for activity-independent human pose inference , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Thomas S. Huang,et al.  Discriminative estimation of 3D human pose using Gaussian processes , 2008, 2008 19th International Conference on Pattern Recognition.

[10]  Carl E. Rasmussen,et al.  Warped Gaussian Processes , 2003, NIPS.

[11]  David J. Fleet,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE Gaussian Process Dynamical Model , 2007 .

[12]  J. Vanhatalo,et al.  Approximate inference for disease mapping with sparse Gaussian processes , 2010, Statistics in medicine.

[13]  Marina Vannucci,et al.  Spiked Dirichlet Process Priors for Gaussian Process Models. , 2010, Journal of probability and statistics.

[14]  Shaogang Gong,et al.  Modelling Multi-object Activity by Gaussian Processes , 2009, BMVC.

[15]  Gavin C. Cawley,et al.  Generalised Kernel Machines , 2007, 2007 International Joint Conference on Neural Networks.

[16]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[17]  Hyun-Chul Kim,et al.  Bayesian Gaussian Process Classification with the EM-EP Algorithm , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Aphrodite Galata,et al.  Local Gaussian Processes for Pose Recognition from Noisy Inputs , 2010, BMVC.

[19]  Zhihua Zhang,et al.  Bayesian Generalized Kernel Models , 2010, AISTATS.

[20]  Cristian Sminchisescu,et al.  Twin Gaussian Processes for Structured Prediction , 2010, International Journal of Computer Vision.

[21]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[22]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[23]  Aude Billard,et al.  Calibration-Free Eye Gaze Direction Detection with Gaussian Processes , 2008, VISAPP.

[24]  Malte Kuß,et al.  Gaussian process models for robust regression, classification, and reinforcement learning , 2006 .

[25]  G. Kitagawa.  Non-Gaussian State—Space Modeling of Nonstationary Time Series , 1987 .

[26]  David J. Fleet,et al.  Priors for people tracking from small training sets , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[27]  Nuno Vasconcelos,et al.  Bayesian Poisson regression for crowd counting , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[28]  Warren B. Powell,et al.  Dirichlet Process Mixtures of Generalized Linear Models , 2009, J. Mach. Learn. Res..

[29]  Matthias W. Seeger,et al.  Convex variational Bayesian inference for large scale generalized linear models , 2009, ICML '09.

[30]  Eric Sommerlade,et al.  Modelling pedestrian trajectory patterns with Gaussian processes , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[31]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[32]  C. Gouriéroux,et al.  Non-Gaussian State-Space Modeling of Nonstationary Time Series , 2008 .

[33]  Antoni B. Chan,et al.  Generalized Gaussian process models , 2011, CVPR 2011.

[34]  Wolfram Burgard,et al.  Gaussian Beam Processes: A Nonparametric Bayesian Measurement Model for Range Finders , 2007, Robotics: Science and Systems.

[35]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[36]  David G. Stork,et al.  Pattern Classification , 1973 .

[37]  Michael R. Lyu,et al.  Nonrigid shape recovery by Gaussian process regression , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  T. Choi,et al.  Gaussian Process Regression Analysis for Functional Data , 2011 .

[39]  Seth D Guikema,et al.  A Flexible Count Data Regression Model for Risk Analysis , 2008, Risk analysis : an official publication of the Society for Risk Analysis.

[40]  Aki Vehtari,et al.  Sparse Log Gaussian Processes via MCMC for Spatial Epidemiology , 2007, Gaussian Processes in Practice.

[41]  Mark J. Schervish,et al.  Nonstationary Covariance Functions for Gaussian Process Regression , 2003, NIPS.

[42]  Ehud Rivlin,et al.  Tracking and Classifying of Human Motions with Gaussian Process Annealed Particle Filter , 2007, ACCV.

[43]  Nuno Vasconcelos,et al.  Privacy preserving crowd monitoring: Counting people without people models or tracking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  L. Fahrmeir,et al.  Multivariate statistical modelling based on generalized linear models , 1994 .

[45]  Matthias Bethge,et al.  Bayesian Inference for Sparse Generalized Linear Models , 2007, ECML.

[46]  C. Biller.  Adaptive Bayesian Regression Splines in Semiparametric Generalized Linear Models , 2000 .

[47]  Mark Girolami,et al.  Variational Bayesian Multinomial Probit Regression with Gaussian Process Priors , 2006, Neural Computation.

[48]  Dit-Yan Yeung,et al.  Multi-task warped Gaussian process for personalized age estimation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[49]  Carl E. Rasmussen,et al.  Gaussian Processes for Machine Learning (GPML) Toolbox , 2010, J. Mach. Learn. Res..

[50]  J. Albert.  Computational methods using a Bayesian hierarchical generalized linear model , 1988 .

[51]  Aki Vehtari,et al.  Robust Gaussian Process Regression with a Student-t Likelihood , 2011, J. Mach. Learn. Res..

[52]  C. Rasmussen,et al.  Approximations for Binary Gaussian Process Classification , 2008 .

[53]  Trevor Darrell,et al.  Gaussian Processes for Object Categorization , 2010, International Journal of Computer Vision.

[54]  Aki Vehtari,et al.  Bayesian Modeling with Gaussian Processes using the MATLAB Toolbox GPstuff (v3.3) , 2012, ArXiv.

[55]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[56]  David Barber,et al.  Bayesian Classification With Gaussian Processes , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Eric R. Ziegel,et al.  Multivariate Statistical Modelling Based on Generalized Linear Models , 2002, Technometrics.

[58]  Yee Whye Teh,et al.  Semiparametric latent factor models , 2005, AISTATS.

[59]  David J. C. MacKay,et al.  Variational Gaussian process classifiers , 2000, IEEE Trans. Neural Networks Learn. Syst..

[60]  Oliver Williams,et al.  A Switched Gaussian Process for Estimating Disparity and Segmentation in Binocular Stereo , 2006, NIPS.

[61]  Volker Tresp,et al.  The generalized Bayesian committee machine , 2000, KDD '00.

[62]  P. Diggle,et al.  Model‐based geostatistics , 2007 .

[63]  Aki Vehtari,et al.  Gaussian process regression with Student-t likelihood , 2009, NIPS.

[64]  Manfred Opper,et al.  The Variational Gaussian Approximation Revisited , 2009, Neural Computation.

[65]  Qiang Ji,et al.  Switching Gaussian Process Dynamic Models for simultaneous composite motion tracking and recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[66]  Carl E. Rasmussen,et al.  Assessing Approximate Inference for Binary Gaussian Process Classification , 2005, J. Mach. Learn. Res..

[67]  D. C. Howell.  Fundamental Statistics for the Behavioral Sciences , 1985 .

[68]  Dong Han,et al.  Selection and context for action recognition , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[69]  D. Dey,et al.  On Bayesian Analysis of Generalized Linear Models : A New Perspective , 2007 .