Generalization Errors and Learning Curves for Regression with Multi-task Gaussian Processes

We provide some insights into how task correlations in multi-task Gaussian process (GP) regression affect the generalization error and the learning curve. We analyze the asymmetric two-tasks case, where a secondary task is to help the learning of a primary task. Within this setting, we give bounds on the generalization error and the learning curve of the primary task. Our approach admits intuitive understandings of the multi-task GP by relating it to single-task GPs. For the case of one-dimensional input-space under optimal sampling with data only for the secondary task, the limitations of multi-task GP can be quantified explicitly.

[1]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[2]  Christopher K. I. Williams,et al.  Upper and Lower Bounds on the Learning Curve for Gaussian Processes , 2000, Machine Learning.

[3]  R. Taylor,et al.  The Numerical Treatment of Integral Equations , 1978 .

[4]  Lawrence Carin,et al.  Multi-Task Learning for Classification with Dirichlet Process Priors , 2007, J. Mach. Learn. Res..

[5]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[6]  Thomas Hofmann,et al.  Stochastic Relational Models for Discriminative Link Prediction , 2007 .

[7]  Rajeev Sharma,et al.  Advances in Neural Information Processing Systems 11 , 1999 .

[8]  Klaus Ritter,et al.  Average-case analysis of numerical problems , 2000, Lecture notes in mathematics.

[9]  Shai Ben-David,et al.  A notion of task relatedness yielding provable multiple-task learning guarantees , 2008, Machine Learning.

[10]  B. Birmingham.  Gaussian Regression and Optimal Finite Dimensional Linear Models , 1997 .

[11]  Yee Whye Teh,et al.  Semiparametric latent factor models , 2005, AISTATS.

[12]  Edwin V. Bonilla,et al.  Kernel Multi-task Learning using Task-specific Features , 2007, AISTATS.

[13]  Jonathan Baxter,et al.  A Model of Inductive Bias Learning , 2000, J. Artif. Intell. Res..

[14]  Manfred Opper,et al.  Finite-Dimensional Approximation of Gaussian Processes , 1998, NIPS.

[15]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[16]  Peter Sollich,et al.  Learning Curves for Gaussian Process Regression: Approximations and Bounds , 2001, Neural Computation.

[17]  Christopher K. I. Williams,et al.  Gaussian regression and optimal finite dimensional linear models , 1997 .

[18]  Andreas Maurer,et al.  Bounds for Linear Multi-Task Learning , 2006, J. Mach. Learn. Res..

[19]  Mike Rees,et al.  5. Statistics for Spatial Data , 1993 .

[20]  Manfred Opper,et al.  General Bounds on Bayes Errors for Regression with Gaussian Processes , 1998, NIPS.