Effectiveness of Random Search in SVM hyper-parameter tuning

Classification is one of the most common machine learning tasks. SVMs have been frequently applied to this task. In general, the values chosen for the hyper-parameters of SVMs affect the performance of their induced predictive models. Several studies use optimization techniques to find a set of hyper-parameter values that induces classifiers with good predictive performance. This paper investigates the hypothesis that a simple Random Search method is sufficient to adjust the hyper-parameters of SVMs. A set of experiments compared the performance of five tuning techniques: three meta-heuristics commonly used, Random Search and Grid Search. The experimental results show that the predictive performance of models using Random Search is equivalent to those obtained using meta-heuristics and Grid Search, but with a lower computational cost.

[1]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Combining meta-learning and search techniques to select parameters for support vector machines , 2012, Neurocomputing.

[2]  B. Lang,et al.  Efficient optimization of support vector machine learning parameters for unbalanced datasets , 2006 .

[3]  Kate Smith-Miles,et al.  A meta-learning approach to automatic kernel selection for support vector machines , 2006, Neurocomputing.

[4]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[5]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[6]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[7]  Sigrún Andradóttir,et al.  A Review of Random Search Methods , 2015 .

[8]  Luís Torgo,et al.  OpenML: A Collaborative Science Platform , 2013, ECML/PKDD.

[9]  Ching Y. Suen,et al.  Optimization of the SVM Kernels Using an Empirical Error Minimization Scheme , 2002, SVM.

[10]  Jason Weston,et al.  A user's guide to support vector machines. , 2010, Methods in molecular biology.

[11]  Carlos Soares,et al.  A Meta-Learning Method to Select the Kernel Width in Support Vector Regression , 2004, Machine Learning.

[12]  Thomas Stützle,et al.  Automatic Algorithm Configuration Based on Local Search , 2007, AAAI.

[13]  Christian Igel,et al.  Evolutionary tuning of multiple SVM parameters , 2005, ESANN.

[14]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Bio-inspired Optimization Techniques for SVM Parameter Tuning , 2008, 2008 10th Brazilian Symposium on Neural Networks.

[15]  Bernd Bischl,et al.  Tuning and evolution of support vector kernels , 2012, Evol. Intell..

[16]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[17]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[18]  James Kennedy.  Particle Swarms: Optimization Based on Sociocognition , 2005 .

[19]  Martin Pelikan,et al.  An introduction and survey of estimation of distribution algorithms , 2011, Swarm Evol. Comput..

[20]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[21]  A. E. Eiben,et al.  Efficient relevance estimation and value calibration of evolutionary algorithm parameters , 2007, 2007 IEEE Congress on Evolutionary Computation.

[22]  Michèle Sebag,et al.  Collaborative hyperparameter tuning , 2013, ICML.

[23]  Oliviero Carugo,et al.  Data Mining Techniques for the Life Sciences , 2009, Methods in Molecular Biology.

[24]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[25]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[26]  Ljubomir J. Buturovic,et al.  Cross-validation pitfalls when selecting and assessing regression and classification models , 2014, Journal of Cheminformatics.

[27]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[28]  M. C. Monard,et al.  A Note on Parameter Selection for Support Vector Machines , 2013, MICAI.