A fast algorithm for kernel 1-norm support vector machines

This paper presents a fast algorithm called Column Generation Newton (CGN) for kernel 1-norm support vector machines (SVMs). CGN combines the Column Generation (CG) algorithm and the Newton Linear Programming SVM (NLPSVM) method. NLPSVM was proposed for solving 1-norm SVM, and CG is frequently used in large-scale integer and linear programming algorithms. In each iteration of the kernel 1-norm SVM, NLPSVM has a time complexity of O(@?^3), where @? is the sample number, and CG has a time complexity between O(@?^3) and O(n^'^3), where n' is the number of columns of the coefficient matrix in the subproblem. CGN uses CG to generate a sequence of subproblems containing only active constraints and then NLPSVM to solve each subproblem. Since the subproblem in each iteration only consists of n' unbound constraints, CGN thus has a time complexity of O(n^'^3), which is smaller than that of NLPSVM and CG. Also, CGN is faster than CG when the solution to 1-norm SVM is sparse. A theorem is given to show a finite step convergence of CGN. Experimental results on the Ringnorm and UCI data sets demonstrate the efficiency of CGN to solve the kernel 1-norm SVM.

[1]  Kristin P. Bennett,et al.  Combining support vector and mathematical programming methods for classification , 1999 .

[2]  Jue Wang,et al.  A general soft method for learning SVM classifiers with L1-norm penalty , 2008, Pattern Recognit..

[3]  Gert R. G. Lanckriet,et al.  Classification of a large microarray data set: algorithm comparison and analysis of drug signatures. , 2005, Genome research.

[4]  Olvi L. Mangasarian,et al.  Generalized Support Vector Machines , 1998 .

[5]  Li Zhang,et al.  Density-induced margin support vector machines , 2011, Pattern Recognit..

[6]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[7]  Jinbo Bi,et al.  Dimensionality Reduction via Sparse Support Vector Machines , 2003, J. Mach. Learn. Res..

[8]  Xiaodong Lin,et al.  Gene expression Gene selection using support vector machines with non-convex penalty , 2005 .

[9]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[10]  Jian-xiong Dong,et al.  Fast SVM training algorithm with decomposition on very large data sets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Ingo Steinwart,et al.  Sparseness of Support Vector Machines , 2003, J. Mach. Learn. Res..

[12]  Federico Girosi,et al.  An improved training algorithm for support vector machines , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[13]  Li Zhang,et al.  Hidden space support vector machines , 2004, IEEE Transactions on Neural Networks.

[14]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[15]  Glenn Fung,et al.  A Feature Selection Newton Method for Support Vector Machine Classification , 2004, Comput. Optim. Appl..

[16]  Shigeo Abe,et al.  Decomposition techniques for training linear programming support vector machines , 2009, Neurocomputing.

[17]  Ayhan Demiriz,et al.  Linear Programming Boosting via Column Generation , 2002, Machine Learning.

[18]  Glenn Fung,et al.  SVM Feature Selection for Classification of SPECT Images of Alzheimer's Disease Using Spatial Information , 2005, ICDM.

[19]  Li Zhang,et al.  On the sparseness of 1-norm support vector machines , 2010, Neural Networks.

[20]  Alexander J. Smola,et al.  Sparse Greedy Gaussian Process Regression , 2000, NIPS.

[21]  Ivor W. Tsang,et al.  Core Vector Machines: Fast SVM Training on Very Large Data Sets , 2005, J. Mach. Learn. Res..

[22]  O. Mangasarian,et al.  Massive data discrimination via linear support vector machines , 2000 .

[23]  Li Zhang,et al.  Linear programming support vector machines , 2002, Pattern Recognit..

[24]  Olvi L. Mangasarian,et al.  Exact 1-Norm Support Vector Machines Via Unconstrained Convex Differentiable Minimization , 2006, J. Mach. Learn. Res..

[25]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[26]  Jacques Desrosiers,et al.  Selected Topics in Column Generation , 2002, Oper. Res..

[27]  Li Zhang,et al.  Hidden Space Principal Component Analysis , 2006, PAKDD.

[28]  Robert J. Vanderbei,et al.  Linear Programming: Foundations and Extensions , 1998, Kluwer international series in operations research and management service.

[29]  Li Zhang,et al.  Decision Tree Support Vector Machine , 2007, Int. J. Artif. Intell. Tools.

[30]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[31]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[32]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.