Quadratic optimization fine tuning for the Support Vector Machines learning phase

This work presents a comparative analysis of specific, rather than general, mathematical programming implementation techniques of the quadratic optimization problem (QP) based on Support Vector Machines (SVM) learning process. Considering the Karush-Kuhn-Tucker (KKT) optimality conditions, we present a strategy of implementation of the SVM-QP following three classical approaches: (i) active set, also divided in primal and dual spaces, methods, (ii) interior point methods and (iii) linearization strategies. We also present the general extension to treat large-scale applications consisting in a general decomposition of the QP problem into smaller ones, conserving the exact solution approach. In the same manner, we propose a set of heuristics to take into account for a better than a random selection process for the initialization of the decomposition strategy. We compare the performances of the optimization strategies using some well-known benchmark databases.

[1]  J. Gondzio,et al.  Regularized Symmetric Indefinite Systems in Interior Point Methods for Linear and Quadratic Optimization , 1999 .

[2]  Zhi-Jie He,et al.  A new fast training algorithm for SVM , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[3]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[4]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[5]  Roger Fletcher,et al.  Practical methods of optimization; (2nd ed.) , 1987 .

[6]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[7]  Robert M. Freund,et al.  Interior point methods : current status and future directions , 1996 .

[8]  Federico Girosi,et al.  Support Vector Machines: Training and Applications , 1997 .

[9]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[10]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[11]  Robert J. Vanderbei,et al.  Linear Programming: Foundations and Extensions , 1998, Kluwer international series in operations research and management service.

[12]  Stephen J. Wright,et al.  Optimization Software Guide , 1987 .

[13]  Juan Frausto Solís,et al.  Increasing the Training Speed of SVM, the Zoutendijk Algorithm Case , 2005, ISSADS.

[14]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[15]  Jacek Gondzio,et al.  Implementation of Interior Point Methods for Large Scale Linear Programming , 1996 .

[16]  Stephen J. Wright,et al.  Object-oriented software for quadratic programming , 2003, TOMS.

[17]  Rodolfo Ibarra-Orozco,et al.  A Comparison of Different Initialization Strategies to Reduce the Training Time of Support Vector Machines , 2005, ICANN.

[18]  Linda Kaufman,et al.  Solving the quadratic programming problem arising in support vector classification , 1999 .

[19]  Donald Goldfarb,et al.  A numerically stable dual method for solving strictly convex quadratic programs , 1983, Math. Program..