Dual active set method for support vector machines under multi-constraint activation

Active set methods are competitive alternatives of SVM optimizers to working set or decomposition techniques for moderate scale problems. While there are many works on applying the primal active set methods to SVM, few study the dual active set methods. Comparing with the primal active set method, the dual active set method is more efficient and numerically stable due to using an unconstrained minimum as a feasible start point and having choices of which constraints to add to the active set. However, since the conventional dual active set method can only add one violated constraint in one step, this paper proposes a novel dual active set method, which allows to add multiple constraints and accelerates the convergence. Moreover, the new method does not require to compute the matrix inversion when applied to SVM, and essentially reduces the training time. Experiment results on several benchmark data sets validate that the proposed method is indeed more efficient than the primal and the conventional dual active set methods.

[1]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[2]  Michael Vogt,et al.  Active-Set Methods for Support Vector Machines , 2005 .

[3]  Shigeo Abe,et al.  Convergence Improvement of Active Set Training for Support Vector Regressors , 2010, ICANN.

[4]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[5]  David R. Musicant,et al.  Active Support Vector Machine Classification , 2000, NIPS.

[6]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[7]  V. Kecman,et al.  Iterative Single Data Algorithm for Training Kernel Machines from Huge Data Sets: Theory and Performance , 2005 .

[8]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[9]  Natashia Boland,et al.  A dual-active-set algorithm for positive semi-definite quadratic programming , 1997, Math. Program..

[10]  Donald Goldfarb,et al.  A numerically stable dual method for solving strictly convex quadratic programs , 1983, Math. Program..

[11]  David R. Musicant,et al.  Active set support vector regression , 2004, IEEE Transactions on Neural Networks.

[12]  J. Platt.  Sequential Minimal Optimization : A Fast Algorithm for Training Support Vector Machines , 1998 .

[13]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[14]  Katya Scheinberg,et al.  An Efficient Implementation of an Active Set Method for SVMs , 2006, J. Mach. Learn. Res..

[15]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[16]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[17]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[18]  Bernhard Schölkopf,et al.  Comparison of View-Based Object Recognition Algorithms Using Realistic 3D Models , 1996, ICANN.

[19]  Karl Kunisch,et al.  Primal-Dual Strategy for State-Constrained Optimal Control Problems , 2002, Comput. Optim. Appl..

[20]  Georgios C. Anagnostopoulos,et al.  Efficient Revised Simplex Method for SVM Training , 2011, IEEE Transactions on Neural Networks.

[21]  Jason Weston,et al.  A user's guide to support vector machines. , 2010, Methods in molecular biology.

[22]  Shigeo Abe,et al.  Convergence improvement of active set support vector training , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[23]  Bernhard Schölkopf,et al.  Improving the Accuracy and Speed of Support Vector Machines , 1996, NIPS.

[24]  M. Hintermueller,et al.  A primal-dual active set algorithm for bilaterally control constrained optimal control problems , 2003 .

[25]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[26]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[27]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[28]  Kazufumi Ito,et al.  The Primal-Dual Active Set Strategy as a Semismooth Newton Method , 2002, SIAM J. Optim..

[29]  Bernhard Schölkopf,et al.  Extracting Support Data for a Given Task , 1995, KDD.

[30]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[31]  Kazufumi Ito,et al.  The Primal-Dual Active Set Method for Nonlinear Optimal Control Problems with Bilateral Constraints , 2004, SIAM J. Control. Optim..

[32]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.