Mean field method for the support vector machine regression

Abstract This paper deals with two subjects. First, we will show how support vector machine (SVM) regression problem can be solved as the maximum a posteriori prediction in the Bayesian framework. The second part describes an approximation technique that is useful in performing calculations for SVMs based on the mean field algorithm which was originally proposed in Statistical Physics of disordered systems. One advantage is that it handle posterior averages for Gaussian process which are not analytically tractable.

[1]  Peter Sollich.  Approximate learning curves for Gaussian processes , 1999 .

[2]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[3]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[4]  Junbin Gao,et al.  A Probabilistic Framework for SVM Regression and Error Bar Estimation , 2002, Machine Learning.

[5]  Christopher K. I. Williams.  Regression with Gaussian processes , 1997 .

[6]  Massimiliano Pontil,et al.  On the Noise Model of Support Vector Machines Regression , 2000, ALT.

[7]  G. Wahba.  Support vector machines, reproducing kernel Hilbert spaces, and randomized GACV , 1999 .

[8]  Tomaso Poggio,et al.  A Unified Framework for Regularization Networks and Support Vector Machines , 1999 .

[9]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[10]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[11]  David Barber,et al.  Bayesian Classification With Gaussian Processes , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  David Barber,et al.  Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo , 1996, NIPS.

[13]  Peter Sollich.  Probabilistic interpretations and Bayesian methods for support vector machines , 1999 .

[14]  Christopher K. I. Williams.  Computing with Infinite Networks , 1996, NIPS.

[15]  R. Vanderbei.  LOQO:an interior point code for quadratic programming , 1999 .

[16]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[17]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[18]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[19]  Ole Winther,et al.  Gaussian Processes for Classification: Mean-Field Algorithms , 2000, Neural Computation.

[20]  L. Galway.  Spline Models for Observational Data , 1991 .

[21]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[23]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[24]  Michael I. Jordan,et al.  Variational Probabilistic Inference and the QMR-DT Network , 2011, J. Artif. Intell. Res..

[25]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[26]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[27]  Christopher K. I. Williams.  Prediction with Gaussian Processes: From Linear Regression to Linear Prediction and Beyond , 1999, Learning in Graphical Models.

[28]  S. Gunn.  Support Vector Machines for Classification and Regression , 1998 .

[29]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[30]  Michael I. Jordan,et al.  Variational probabilistic inference and the QMR-DT database , 1998 .

[31]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[32]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .