Genetic Algorithms for Support Vector Machine Model Selection

The support vector machine is a powerful classifier that has been successfully applied to a broad range of pattern recognition problems in various domains, e.g. corporate decision making, text and image recognition or medical diagnosis. Support vector machines belong to the group of semiparametric classifiers. The selection of appropriate parameters, formally known as model selection, is crucial to obtain accurate classification results for a given task. Striving to automate model selection for support vector machines we apply a meta-strategy utilizing genetic algorithms to learn combined kernels in a data-driven manner and to determine all free kernel parameters. The model selection criterion is incorporated into a fitness function guiding the evolutionary process of classifier construction. We consider two types of criteria consisting of empirical estimators or theoretical bounds for the generalization error. We evaluate their effectiveness in an empirical study on four well known benchmark data sets to find that both are applicable fitness measures for constructing accurate classifiers and conducting model selection. However, model selection focuses on finding one best classifier while genetic algorithms are based on the idea of re-combining and mutating a large number of good candidate classifiers to realize further improvements. It is shown that the empirical estimator is the superior fitness criterion in this sense, leading to a greater number of promising models on average.

[1]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[2]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[3]  Thorsten Joachims,et al.  Estimating the Generalization Performance of an SVM Efficiently , 2000, ICML.

[4]  Dong Seong Kim,et al.  Determining Optimal Decision Model for Support Vector Machine by Genetic Algorithm , 2004, CIS.

[5]  Olivier Chapelle,et al.  Model Selection for Support Vector Machines , 1999, NIPS.

[6]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[7]  Chih-Jen Lin,et al.  Radius Margin Bounds for Support Vector Machines with the RBF Kernel , 2002, Neural Computation.

[8]  Johan A. K. Suykens,et al.  Knowledge discovery in a direct marketing case using least squares support vector machines , 2001, Int. J. Intell. Syst..

[9]  Siddhartha Bhattacharyya,et al.  Direct Marketing Response Models Using Genetic Algorithms , 1998, KDD.

[10]  B. Samanta,et al.  Gear fault detection using artificial neural networks and support vector machines with genetic algorithms , 2004 .

[11]  K. Johana,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2022 .

[12]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[13]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[14]  V. Vapnik.  Estimation of Dependences Based on Empirical Data , 2006 .

[15]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[16]  Johan A. K. Suykens,et al.  Benchmarking state-of-the-art classification algorithms for credit scoring , 2003, J. Oper. Res. Soc..

[17]  Christian Igel,et al.  Evolutionary tuning of multiple SVM parameters , 2005, ESANN.

[18]  Nozha Boujemaa,et al.  The LCCP for Optimizing Kernel Parameters for SVM , 2005, ICANN.

[19]  S. Sathiya Keerthi,et al.  Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms , 2002, IEEE Trans. Neural Networks.

[20]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[21]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[22]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[23]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[24]  Ching Y. Suen,et al.  Automatic model selection for the optimization of SVM kernels , 2005, Pattern Recognit..

[25]  Yanxi Liu,et al.  Cervical Cancer Detection Using SVM Based Feature Screening , 2004, MICCAI.

[26]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[27]  Sung-Do Chi,et al.  Evolutionary Parameter Estimation Algorithm for Combined Kernel Function in Support Vector Machine , 2004, AWCC.

[28]  S. Sathiya Keerthi,et al.  Evaluation of simple performance measures for tuning SVM hyperparameters , 2003, Neurocomputing.

[29]  Ha-Nam Nguyen,et al.  Combined Kernel Function for Support Vector Machine and Learning Method Based on Evolutionary Algorithm , 2004, ICONIP.

[30]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[31]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[32]  Li Li,et al.  A robust hybrid between genetic algorithm and support vector machine for extracting an optimal feature gene subset. , 2005, Genomics.