Multi-task Sparse Gaussian Processes with Improved Multi-task Sparsity Regularization

Gaussian processes are a popular and effective Bayesian method for classification and regression. Generating sparse Gaussian processes is a hot research topic, since Gaussian processes have to face the problem of cubic time complexity with respect to the size of the training set. Inspired by the idea of multi-task learning, we believe that simultaneously selecting subsets of multiple Gaussian processes will be more suitable than selecting them separately. In this paper, we propose an improved multi-task sparsity regularizer which can effectively regularize the subset selection of multiple tasks for multi-task sparse Gaussian processes. In particular, based on the multi-task sparsity regularizer proposed in [12], we perform two improvements: 1) replacing a subset of points with a rough global structure when measuring the global consistency of one point; 2) performing normalization on each dimension of every data set before sparsification. We combine the regularizer with two methods to demonstrate its effectiveness. Experimental results on four real data sets show its superiority.

[1]  Tony Jebara,et al.  Multitask Sparsity via Maximum Entropy Discrimination , 2011, J. Mach. Learn. Res..

[2]  Shiliang Sun,et al.  Infinite mixtures of multivariate Gaussian processes , 2013, 2013 International Conference on Machine Learning and Cybernetics.

[3]  Edwin V. Bonilla,et al.  Kernel Multi-task Learning using Task-specific Features , 2007, AISTATS.

[4]  Shiliang Sun.  Multitask learning for EEG-based biometrics , 2008, 2008 19th International Conference on Pattern Recognition.

[5]  Shiliang Sun,et al.  Manifold-preserving graph reduction for sparse semi-supervised learning , 2014, Neurocomputing.

[6]  Peter A. Flach,et al.  Evaluation Measures for Multi-class Subgroup Discovery , 2009, ECML/PKDD.

[7]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[8]  Shiliang Sun,et al.  Sparse Gaussian processes with manifold-preserving graph reduction , 2014, Neurocomputing.

[9]  Roni Khardon,et al.  Sparse Gaussian Processes for Multi-task Learning , 2012, ECML/PKDD.

[10]  Shiliang Sun,et al.  Single-task and multitask sparse Gaussian processes , 2013, 2013 International Conference on Machine Learning and Cybernetics.

[11]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[12]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[13]  Dean P. Foster,et al.  Minimum Description Length Penalization for Group and Multi-Task Sparse Learning , 2011, J. Mach. Learn. Res..

[14]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[15]  Neil D. Lawrence,et al.  Learning to learn with the informative vector machine , 2004, ICML.