Simple Proof of Convergence of the SMO Algorithm for Different SVM Variants

In this brief, we give a new proof of the asymptotic convergence of the sequential minimum optimization (SMO) algorithm for both the most violating pair and second order rules to select the pair of coefficients to be updated. The proof is more self-contained, shorter, and simpler than previous ones and has a different flavor, partially building upon Gilbert's original convergence proof of its algorithm to solve the minimum norm problem for convex hulls. It is valid for both support vector classification (SVC) and support vector regression, which are formulated under a general problem that encompasses them. Moreover, this general problem can be further extended to also cover other support vector machines (SVM)-related problems such as -SVC or one-class SVMs, while the convergence proof of the slight variant of SMO needed for them remains basically unchanged.

[1]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[2]  Norikazu Takahashi,et al.  Rigorous proof of termination of SMO algorithm for support vector Machines , 2005, IEEE Transactions on Neural Networks.

[3]  S. Sathiya Keerthi,et al.  Improvements to the SMO algorithm for SVM regression , 2000, IEEE Trans. Neural Networks Learn. Syst..

[4]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[5]  Hans Ulrich Simon,et al.  SVM-Optimization and Steepest-Descent Line Search , 2009, COLT.

[6]  Chih-Jen Lin,et al.  Asymptotic convergence of an SMO algorithm without any assumptions , 2002, IEEE Trans. Neural Networks.

[7]  S. Sathiya Keerthi,et al.  Convergence of a Generalized SMO Algorithm for SVM Classifier Design , 2002, Machine Learning.

[8]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[9]  Thorsten Joachims,et al.  Making large-scale support vector machine learning practical , 1999 .

[10]  Chih-Jen Lin,et al.  A Study on SMO-Type Decomposition Methods for Support Vector Machines , 2006, IEEE Transactions on Neural Networks.

[11]  Jun Guo,et al.  Global Convergence of SMO Algorithm for Support Vector Regression , 2008, IEEE Transactions on Neural Networks.

[12]  José R. Dorronsoro,et al.  Convergence of algorithms for solving the Nearest Point Problem in Reduced Convex Hulls , 2011, The 2011 International Joint Conference on Neural Networks.

[13]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[14]  Chih-Jen Lin,et al.  On the convergence of the decomposition method for support vector machines , 2001, IEEE Trans. Neural Networks.

[15]  David J. Crisp,et al.  Uniqueness theorems for kernel methods , 2003, Neurocomputing.

[16]  José R. Dorronsoro,et al.  Clipping algorithms for solving the nearest point problem over reduced convex hulls , 2011, Pattern Recognit..

[17]  E. Gilbert.  An Iterative Procedure for Computing the Minimum of a Quadratic Form on a Convex Set , 1966 .

[18]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[19]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[20]  Gary William Flake,et al.  Efficient SVM Regression Training with SMO , 2002, Machine Learning.