An Efficient Support Vector Machine Learning Method with Second-Order Cone Programming for Large-Scale Problems

In this paper we propose a new fast learning algorithm for the support vector machine (SVM). The proposed method is based on the technique of second-order cone programming. We reformulate the SVM's quadratic programming problem into the second-order cone programming problem. The proposed method needs to decompose the kernel matrix of SVM's optimization problem, and the decomposed matrix is used in the new optimization problem. Since the kernel matrix is positive semidefinite, the dimension of the decomposed matrix can be reduced by decomposition (factorization) methods. The performance of the proposed method depends on the dimension of the decomposed matrix. Experimental results show that the proposed method is much faster than the quadratic programming solver LOQO if the dimension of the decomposed matrix is small enough compared to that of the kernel matrix. The proposed method is also faster than the method proposed in (S. Fine and K. Scheinberg, 2001) for both low-rank and full-rank kernel matrices. The working set selection is an important issue in the SVM decomposition (chunking) method. We also modify Hsu and Lin's working set selection approach to deal with large working set. The proposed approach leads to faster convergence.

[1]  Alexander J. Smola,et al.  Support Vector Machine Reference Manual , 1998 .

[2]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[3]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[4]  Samy Bengio,et al.  SVMTorch: Support Vector Machines for Large-Scale Regression Problems , 2001, J. Mach. Learn. Res..

[5]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[6]  Pavel Laskov,et al.  An Improved Decomposition Algorithm for Regression Support Vector Machines , 1999, NIPS.

[7]  Jochen Werner,et al.  Optimization Theory and Applications , 1984 .

[8]  H. Takahashi,et al.  An improved working set selection method for SVM decomposition method , 2004, 2004 2nd International IEEE Conference on 'Intelligent Systems'. Proceedings (IEEE Cat. No.04EX791).

[9]  Renato D. C. Monteiro,et al.  Primal-Dual Path-Following Algorithms for Semidefinite Programming , 1997, SIAM J. Optim..

[10]  LinChih-Jen,et al.  Training -Support Vector Classifiers: Theory and Algorithms , 2001 .

[11]  Pavel Laskov,et al.  Feasible Direction Decomposition Algorithms for Training Support Vector Machines , 2002, Machine Learning.

[12]  Chih-Jen Lin,et al.  Training v-Support Vector Classifiers: Theory and Algorithms , 2001, Neural Computation.

[13]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[14]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[15]  Chih-Jen Lin,et al.  On the convergence of the decomposition method for support vector machines , 2001, IEEE Trans. Neural Networks.

[16]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[17]  Robert J. Vanderbei,et al.  An Interior-Point Method for Semidefinite Programming , 1996, SIAM J. Optim..

[18]  Sanjay Mehrotra,et al.  On the Implementation of a Primal-Dual Interior Point Method , 1992, SIAM J. Optim..

[19]  Chih-Jen Lin,et al.  A Simple Decomposition Method for Support Vector Machines , 2002, Machine Learning.

[20]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[21]  M. Muramatsu,et al.  The support vector machine learning using the second order cone programming , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[22]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[23]  Takashi Tsuchiya,et al.  Polynomial convergence of primal-dual algorithms for the second-order cone program based on the MZ-family of directions , 2000, Math. Program..

[24]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[25]  Shinji Hara,et al.  Interior-Point Methods for the Monotone Semidefinite Linear Complementarity Problem in Symmetric Matrices , 1997, SIAM J. Optim..

[26]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[27]  Erling D. Andersen,et al.  On implementing a primal-dual interior-point method for conic quadratic optimization , 2003, Math. Program..

[28]  R. Vanderbei.  LOQO:an interior point code for quadratic programming , 1999 .

[29]  Nello Cristianini,et al.  Simple Learning Algorithms for Training Support Vector Machines , 1998 .

[30]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[31]  John Platt,et al.  Fast training of svms using sequential minimal optimization , 1998 .

[32]  M. Muramatsu.  On a Commutative Class of Search Directions for Linear Programming over Symmetric Cones , 2002 .