Evolutionary tuning of multiple SVM parameters

The problem of model selection for support vector machines (SVMs) is considered. We propose an evolutionary approach to determine multiple SVM hyperparameters: The covariance matrix adaptation evolution strategy (CMA-ES) is used to determine the kernel from a parameterized kernel space and to control the regularization. Our method is applicable to optimize non-differentiable kernel functions and arbitrary model selection criteria. We demonstrate on benchmark datasets that the CMA-ES improves the results achieved by grid search already when applied to few hyperparameters. Further, we show that the CMA-ES is able to handle much more kernel parameters compared to grid-search and that tuning of the scaling and the rotation of Gaussian kernels can lead to better results in comparison to standard Gaussian kernels with a single bandwidth parameter. In particular, more flexibility of the kernel can reduce the number of support vectors.

[1]  Helge J. Ritter,et al.  Discriminative Densities from Maximum Contrast Estimation , 2002, NIPS.

[2]  Saturnino Maldonado-Bascón,et al.  Model Selection for Support Vector Machines Using Ant Colony Optimization in an Electronic Nose Application , 2006, ANTS Workshop.

[3]  David G. Stork,et al.  Evolution and Learning in Neural Networks , 1990, NIPS.

[4]  Carl Gold,et al.  Model selection for support vector machine classification , 2002, Neurocomputing.

[5]  Jens Jägersküpper,et al.  Analysis of a Simple Evolutionary Algorithm for Minimization in Euclidean Spaces , 2003, ICALP.

[6]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[7]  Xin Yao,et al.  Fast Evolution Strategies , 1997, Evolutionary Programming.

[8]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[9]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[10]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[12]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[13]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  Hans-Georg Beyer,et al.  The Theory of Evolution Strategies , 2001, Natural Computing Series.

[16]  Günter Rudolph,et al.  On Correlated Mutations in Evolution Strategies , 1992, PPSN.

[17]  Elena Marchiori,et al.  Analysis of Proteomic Pattern Data for Cancer Detection , 2004, EvoWorkshops.

[18]  N. Hansen,et al.  Convergence Properties of Evolution Strategies with the Derandomized Covariance Matrix Adaptation: T , 1997 .

[19]  Simon J. Perkins,et al.  Genetic Algorithms and Support Vector Machines for Time Series Classification , 2002, Optics + Photonics.

[20]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[21]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[22]  Hans-Paul Schwefel,et al.  Evolution and Optimum Seeking: The Sixth Generation , 1993 .

[23]  Ingo Rechenberg,et al.  Evolutionsstrategie '94 , 1994, Werkstatt Bionik und Evolutionstechnik.

[24]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[25]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.

[26]  Bernhard Schölkopf,et al.  Feature selection for support vector machines by means of genetic algorithm , 2003, Proceedings. 15th IEEE International Conference on Tools with Artificial Intelligence.

[27]  Ronald M. Summers,et al.  Feature selection for computer-aided polyp detection using genetic algorithms , 2003, SPIE Medical Imaging.

[28]  Chih-Jen Lin,et al.  Radius Margin Bounds for Support Vector Machines with the RBF Kernel , 2002, Neural Computation.

[29]  S. Sathiya Keerthi,et al.  Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms , 2002, IEEE Trans. Neural Networks.