Gaussian Process Learning: A Divide-and-Conquer Approach

The Gaussian Process (GP) model is used widely in many hard machine learning tasks. In practice, it faces the challenge from scalability concerns. In this manuscript, we proposed a domain decomposition method in GP learning. It is shown that the GP model itself has the inherent capability of being trained through divide-and-conquer. Given a large GP learning problem, it can be divided into smaller problems. By solving the smaller problems and merging the solutions, it is guaranteed to reach the solution to the original problem. We further verified the efficiency and the effectiveness of the algorithm through experiments.

[1]  R. Schaback,et al.  Numerical Techniques Based on Radial Basis Functions , 2000 .

[2]  Kennan T. Smith,et al.  Practical and mathematical aspects of the problem of reconstructing objects from radiographs , 1977 .

[3]  Tomaso Poggio,et al.  Everything old is new again: a fresh look at historical approaches in machine learning , 2002 .

[4]  Lehel Csató,et al.  Sparse On-Line Gaussian Processes , 2002, Neural Computation.

[5]  Kwong-Sak Leung,et al.  Generalizing the Bias Term of Support Vector Machines , 2007, IJCAI.

[6]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[7]  Larry S. Davis,et al.  Efficient Kernel Machines Using the Improved Fast Gauss Transform , 2004, NIPS.

[8]  Gene H. Golub,et al.  Matrix computations , 1983 .

[9]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[10]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[11]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[12]  Kwong-Sak Leung,et al.  Large-scale RLSC learning without agony , 2007, ICML '07.

[13]  Iain Murray,et al.  A framework for evaluating approximation methods for Gaussian process regression , 2012, J. Mach. Learn. Res..

[14]  Matthias W. Seeger,et al.  Gaussian Processes For Machine Learning , 2004, Int. J. Neural Syst..

[15]  D. Mackay,et al.  Introduction to Gaussian processes , 1998 .

[16]  A. C. Faul,et al.  Proof of convergence of an iterative technique for thin plate spline interpolation in two dimensions , 1999, Adv. Comput. Math..

[17]  Kwong-Sak Leung,et al.  Generalized Regularized Least-Squares Learning with Predefined Features in a Hilbert Space , 2006, NIPS.

[18]  Cristian Sminchisescu,et al.  Greedy Block Coordinate Descent for Large Scale Gaussian Process Regression , 2008, UAI.

[19]  J. Neumann.  Mathematical Foundations of Quantum Mechanics , 1955 .