Optimal reduction of solutions for support vector machines

Being a universal learning machine, a support vector machine (SVM) suffers from expensive computational cost in the test phase due to the large number of support vectors, and greatly impacts its practical use. To address this problem, we proposed an adaptive genetic algorithm to optimally reduce the solutions for an SVM by selecting vectors from the trained support vector solutions, such that the selected vectors best approximate the original discriminant function. Our method can be applied to SVMs using any general kernel. The size of the reduced set can be used adaptively based on the requirement of the tasks. As such the generalization/complexity trade-off can be controlled directly. The lower bound of the number of selected vectors required to recover the original discriminant function can also be determined.

[1]  A. Ruszczynski,et al.  Nonlinear Optimization , 2006 .

[2]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[3]  J. Platt.  Sequential Minimal Optimization : A Fast Algorithm for Training Support Vector Machines , 1998 .

[4]  T. Nishi,et al.  A learning algorithm for improving the classification speed of support vector machines , 2005, Proceedings of the 2005 European Conference on Circuit Theory and Design, 2005..

[5]  Probal Chaudhuri,et al.  On The Use of Genetic Algorithm with Elitism in Robust and Nonparametric Multivariate Analysis , 2003 .

[6]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[7]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[8]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[9]  Cheng-Lin Liu,et al.  Handwritten digit recognition: benchmarking of state-of-the-art techniques , 2003, Pattern Recognit..

[10]  D. E. Goldberg,et al.  Optimization and Machine Learning , 2022 .

[11]  Thorsten Joachims,et al.  Estimating the Generalization Performance of an SVM Efficiently , 2000, ICML.

[12]  Frank Weber,et al.  Optimal Reduced-Set Vectors for Support Vector Machines with a Quadratic Kernel , 2004, Neural Computation.

[13]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[14]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[15]  Tom Downs,et al.  Exact Simplification of Support Vector Solutions , 2002, J. Mach. Learn. Res..

[16]  John Shawe-Taylor,et al.  Generalisation Error Bounds for Sparse Linear Classifiers , 2000, COLT.

[17]  Harris Drucker,et al.  Learning algorithms for classification: A comparison on handwritten digit recognition , 1995 .

[18]  Qing Li,et al.  Adaptive simplification of solution for support vector machine , 2007, Pattern Recognit..

[19]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.