Gaussian Process Regression Networks

We introduce a new regression framework, Gaussian process regression networks (GPRN), which combines the structural properties of Bayesian neural networks with the nonparametric exibility of Gaussian processes. GPRN accommodates input (predictor) dependent signal and noise correlations between multiple output (response) variables, input dependent length-scales and amplitudes, and heavy-tailed predictive distributions. We derive both elliptical slice sampling and variational Bayes inference procedures for GPRN. We apply GPRN as a multiple output regression and multivariate volatility model, demonstrating substantially improved performance over eight popular multiple output (multi-task) Gaussian process models and three multivariate volatility models on real datasets, including a 1000 dimensional gene expression dataset.

[1]  R. Engle.  New Frontiers for Arch Models , 2002 .

[2]  Milton Abramowitz,et al.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , 1964 .

[3]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[4]  Hans Wackernagel,et al.  Multivariate Geostatistics: An Introduction with Applications , 1996 .

[5]  David Nicholson.  Introduction to Variational Methods , 2008 .

[6]  Aki Vehtari,et al.  Gaussian process regression with Student-t likelihood , 2009, NIPS.

[7]  Ryan P. Adams,et al.  Gaussian process product models for nonparametric nonstationarity , 2008, ICML '08.

[8]  Charles G. Renfro,et al.  Benchmarks and software standards: A case study of GARCH procedures , 1998 .

[9]  Ryan P. Adams,et al.  Incorporating side information into probabilistic matrix factorization using Gaussian Processes , 2010 .

[10]  E. Furlong,et al.  Combinatorial binding predicts spatio-temporal cis-regulatory activity , 2009, Nature.

[11]  A. O'Hagan,et al.  On Outlier Rejection Phenomena in Bayes Inference , 1979 .

[12]  Andrew Gordon Wilson,et al.  Copula Processes , 2010, NIPS.

[13]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[14]  N. Shephard,et al.  Analysis of high dimensional multivariate stochastic volatility models , 2006 .

[15]  Ronald P. Barry,et al.  Blackbox Kriging: Spatial Prediction Without Specifying Variogram Models , 1996 .

[16]  Ryan P. Adams,et al.  Elliptical slice sampling , 2009, AISTATS.

[17]  C. Gouriéroux,et al.  The Wishart Autoregressive Process of Multivariate Stochastic Volatility , 2009 .

[18]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[19]  Wolfram Burgard,et al.  Most likely heteroscedastic Gaussian process regression , 2007, ICML '07.

[20]  Iain Murray.  Introduction To Gaussian Processes , 2008 .

[21]  Manfred Opper,et al.  Sparse Representation for Gaussian Process Models , 2000, NIPS.

[22]  Miguel Lázaro-Gredilla,et al.  Variational Heteroscedastic Gaussian Process Regression , 2011, ICML.

[23]  C. Brownlees,et al.  A Practical Guide to Volatility Forecasting through Calm and Storm , 2011 .

[24]  Neil D. Lawrence,et al.  Computationally Efficient Convolved Multiple Output Gaussian Processes , 2011, J. Mach. Learn. Res..

[25]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[26]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[27]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[28]  M. Ashburner,et al.  Systematic determination of patterns of gene expression during Drosophila embryogenesis , 2002, Genome Biology.

[29]  R. Engle,et al.  Multivariate Simultaneous Generalized ARCH , 1995, Econometric Theory.

[30]  Carl E. Rasmussen,et al.  Assessing Approximate Inference for Binary Gaussian Process Classification , 2005, J. Mach. Learn. Res..

[31]  Christopher M. Bishop,et al.  Regression with Input-Dependent Noise: A Bayesian Treatment , 1996, NIPS.

[32]  Timothy C. Coburn,et al.  Geostatistics for Natural Resources Evaluation , 2000, Technometrics.

[33]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[34]  R. Mazo.  On the theory of brownian motion , 1973 .

[35]  N. Shephard,et al.  Multivariate stochastic variance models , 1994 .

[36]  Ronald P. Barry,et al.  Constructing and fitting models for cokriging and multivariable spatial prediction , 1998 .

[37]  Yee Whye Teh,et al.  Semiparametric latent factor models , 2005, AISTATS.

[38]  H. Wackernagle,et al.  Multivariate geostatistics: an introduction with applications , 1998 .

[39]  Sarvapali D. Ramchurn,et al.  2008 International Conference on Information Processing in Sensor Networks Towards Real-Time Information Processing of Sensor Network Data using Computationally Efficient Multi-output Gaussian Processes , 2022 .

[40]  Paul W. Goldberg,et al.  Regression with Input-dependent Noise: A Gaussian Process Treatment , 1997, NIPS.

[41]  P. Hansen,et al.  A Forecast Comparison of Volatility Models: Does Anything Beat a Garch(1,1)? , 2004 .

[42]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[43]  Ser-Huang Poon,et al.  Practical Issues in Forecasting Volatility , 2005 .

[44]  J. Wooldridge,et al.  A Capital Asset Pricing Model with Time-Varying Covariances , 1988, Journal of Political Economy.

[45]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[46]  D. Owen.  Handbook of Mathematical Functions with Formulas , 1965 .

[47]  P. Mazur.  On the theory of brownian motion , 1959 .

[48]  Marcus R. Frean,et al.  Dependent Gaussian Processes , 2004, NIPS.

[49]  C. F. Sirmans,et al.  Nonstationary multivariate process modeling through spatially varying coregionalization , 2004 .

[50]  Nir Friedman,et al.  Gaussian Process Networks , 2000, UAI.

[51]  A. Dawid.  Posterior expectations for large observations , 1973 .

[52]  Richard E. Turner,et al.  Modeling Natural Sounds with Modulation Cascade Processes , 2007, NIPS.

[53]  Andrew Gordon Wilson,et al.  Generalised Wishart Processes , 2010, UAI.

[54]  Chris Brooks,et al.  Benchmarks and the accuracy of GARCH model estimation , 2001 .

[55]  A. O'Hagan,et al.  Bayesian inference for non‐stationary spatial covariance structure via spatial deformations , 2003 .

[56]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[57]  Neil D. Lawrence,et al.  Sparse Convolved Gaussian Processes for Multi-output Regression , 2008, NIPS.

[58]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[59]  Richard E. Turner.  Statistical models for natural sounds , 2010 .

[60]  Oliver Pfaffel.  Wishart Processes , 2012, 1201.3256.

[61]  Bruno De Finetti,et al.  The Bayesian Approach to the Rejection of Outliers , 1961 .

[62]  John P. Cunningham,et al.  Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity , 2008, NIPS.

[63]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[64]  Geoffrey E. Hinton,et al.  Evaluation of Gaussian processes and other methods for non-linear regression , 1997 .