Computationally Efficient Convolved Multiple Output Gaussian Processes

Recently there has been an increasing interest in regression methods that deal with multiple outputs. This has been motivated partly by frameworks like multitask learning, multisensor networks or structured output data. From a Gaussian processes perspective, the problem reduces to specifying an appropriate covariance function that, whilst being positive semi-definite, captures the dependencies between all the data points and across all the outputs. One approach to account for non-trivial correlations between outputs employs convolution processes. Under a latent function interpretation of the convolution transform we establish dependencies between output variables. The main drawbacks of this approach are the associated computational and storage demands. In this paper we address these issues. We present different efficient approximations for dependent output Gaussian processes constructed through the convolution formalism. We exploit the conditional independencies present naturally in the model. This leads to a form of the covariance similar in spirit to the so called PITC and FITC approximations for a single output. We show experimental results with synthetic and real data, in particular, we show results in school exams score prediction, pollution prediction and gene expression data.

[1]  H. Goldstein,et al.  Differential school effectiveness , 1989 .

[2]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[3]  H. Goldstein Multilevel Modelling of Survey Data , 1991 .

[4]  M. Goulard,et al.  Linear coregionalization model: Tools for estimation and choice of cross-variogram matrix , 1992 .

[5]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[6]  Robert Haining,et al.  Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95 , 1993 .

[7]  Mike Rees,et al.  5. Statistics for Spatial Data , 1993 .

[8]  N. Cressie,et al.  Universal cokriging under intrinsic coregionalization , 1994 .

[9]  Ronald P. Barry,et al.  Blackbox Kriging: Spatial Prediction Without Specifying Variogram Models , 1996 .

[10]  Ronald P. Barry,et al.  Constructing and fitting models for cokriging and multivariable spatial prediction , 1998 .

[11]  David Higdon,et al.  Non-Stationary Spatial Modeling , 2022, 2212.08043.

[12]  Manfred Opper,et al.  Sparse Representation for Gaussian Process Models , 2000, NIPS.

[13]  Timothy C. Coburn,et al.  Geostatistics for Natural Resources Evaluation , 2000, Technometrics.

[14]  Tom Heskes,et al.  Empirical Bayes for Learning to Learn , 2000, ICML.

[15]  Alexander J. Smola,et al.  Sparse Greedy Gaussian Process Regression , 2000, NIPS.

[16]  Christopher K. I. Williams,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[17]  Marc G. Genton,et al.  Classes of Kernels for Machine Learning: A Statistics Perspective , 2002, J. Mach. Learn. Res..

[18]  M. Fuentes Interpolation of nonstationary air pollution processes: a spatial spectral approach , 2002 .

[19]  C. Wikle A kernel-based spectral model for non-Gaussian spatio-temporal processes , 2002 .

[20]  M. Fuentes Spectral methods for nonstationary spatial processes , 2002 .

[21]  D. Higdon Space and Space-Time Modeling using Process Convolutions , 2002 .

[22]  M. Ashburner,et al.  Systematic determination of patterns of gene expression during Drosophila embryogenesis , 2002, Genome Biology.

[23]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[24]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[25]  Christopher K. Wikle,et al.  Hierarchical Bayesian Models for Predicting The Spread of Ecological Processes , 2003 .

[26]  Mark J. Schervish,et al.  Nonstationary Covariance Functions for Gaussian Process Regression , 2003, NIPS.

[27]  Tom Heskes,et al.  Task Clustering and Gating for Bayesian Multitask Learning , 2003, J. Mach. Learn. Res..

[28]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[29]  Marcus R. Frean,et al.  Dependent Gaussian Processes , 2004, NIPS.

[30]  David Higdon,et al.  A process-convolution approach to modelling temperatures in the North Atlantic Ocean , 1998, Environmental and Ecological Statistics.

[31]  L. M. Berliner,et al.  Hierarchical Bayesian space-time models , 1998, Environmental and Ecological Statistics.

[32]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[33]  Charles A. Micchelli,et al.  Learning Multiple Tasks with Kernel Methods , 2005, J. Mach. Learn. Res..

[34]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[35]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[36]  Yee Whye Teh,et al.  Semiparametric latent factor models , 2005, AISTATS.

[37]  Neil D. Lawrence,et al.  Modelling transcriptional regulation using Gaussian Processes , 2006, NIPS.

[38]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[39]  Neil D. Lawrence,et al.  Learning for Larger Datasets with the Gaussian Process Latent Variable Model , 2007, AISTATS.

[40]  Noel A Cressie,et al.  Some topics in convolution-based spatial modeling , 2007 .

[41]  S. Mukherjee,et al.  Nonparametric Bayesian Kernel Models , 2007 .

[42]  Stephen J. Roberts,et al.  Gaussian Processes for Prediction , 2007 .

[43]  Lawrence Carin,et al.  Multi-Task Learning for Classification with Dirichlet Process Priors , 2007, J. Mach. Learn. Res..

[44]  Catherine A. Calder,et al.  Dynamic factor process convolution models for multivariate space–time data with application to air quality assessment , 2007, Environmental and Ecological Statistics.

[45]  Sayan Mukherjee,et al.  Characterizing the Function Space for Bayesian Kernel Models , 2007, J. Mach. Learn. Res..

[46]  Zoubin Ghahramani,et al.  Local and global sparse Gaussian process approximations , 2007, AISTATS.

[47]  Neil D. Lawrence,et al.  Sparse Convolved Gaussian Processes for Multi-output Regression , 2008, NIPS.

[48]  Sarvapali D. Ramchurn,et al.  2008 International Conference on Information Processing in Sensor Networks Towards Real-Time Information Processing of Sensor Network Data using Computationally Efficient Multi-output Gaussian Processes , 2022 .

[49]  Sethu Vijayakumar,et al.  Multi-task Gaussian Process Learning of Robot Inverse Dynamics , 2008, NIPS.

[50]  Neil D. Lawrence,et al.  Gaussian process modelling of latent chemical species: applications to inferring transcription factor activities , 2008, ECCB.

[51]  E. Furlong,et al.  Combinatorial binding predicts spatio-temporal cis-regulatory activity , 2009, Nature.

[52]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[53]  Neil D. Lawrence,et al.  Latent Force Models , 2009, AISTATS.

[54]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[55]  Antti Honkela,et al.  Model-based method for transcription factor target identification with limited data , 2010, Proceedings of the National Academy of Sciences.

[56]  Bernhard Schölkopf,et al.  Switched Latent Force Models for Movement Segmentation , 2010, NIPS.

[57]  Neil D. Lawrence,et al.  Efficient Multioutput Gaussian Processes through Variational Inducing Kernels , 2010, AISTATS.

[58]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions , 2012 .

[59]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions: a Review , 2011, Found. Trends Mach. Learn..