The analysis of decomposition methods for support vector machines

The support vector machine (SVM) is a new and promising technique for pattern recognition. It requires the solution of a large dense quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, very few methods can handle the memory problem and an important one is the "decomposition method." However, there is no convergence proof so far. In this paper, we connect this method to projected gradient methods and provide theoretical proofs for a version of decomposition methods. An extension to bound-constrained formulation of SVM is also provided.We then show that this convergence proof is valid for general decomposition methods if their working set selection meets a simple requirement.

[1]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[2]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[3]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[4]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[5]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[7]  Nello Cristianini,et al.  The Kernel-Adatron Algorithm: A Fast and Simple Learning Procedure for Support Vector Machines , 1998, ICML.

[8]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[9]  Alexander J. Smola,et al.  Support Vector Machine Reference Manual , 1998 .

[10]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[11]  David R. Musicant,et al.  Successive overrelaxation for support vector machines , 1999, IEEE Trans. Neural Networks.

[12]  B. Schölkopf,et al.  Advances in kernel methods: support vector learning , 1999 .

[13]  Linda Kaufman,et al.  Solving the quadratic programming problem arising in support vector classification , 1999 .

[14]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.