Optimal Hyper-Parameter Tuning of SVM Classifiers With Application to Medical Diagnosis

Proper tuning of hyper-parameters is essential to the successful application of SVM-classifiers. Several methods have been used for this problem: grid search, random search, estimation of distribution Algorithms (EDAs), bio-inspired metaheuristics, among others. The objective of this paper is to determine the optimal method among those that recently reported good results: Bat algorithm, Firefly algorithm, Fruit-fly optimization algorithm, particle Swarm optimization, Univariate Marginal Distribution Algorithm (UMDA), and Boltzmann-UMDA. The criteria for optimality include measures of effectiveness, generalization, efficiency, and complexity. Experimental results on 15 medical diagnosis problems reveal that EDAs are the optimal strategy under such criteria. Finally, a novel performance index to guide the optimization process, that improves the generalization of the solutions while maintaining their effectiveness, is presented.

[1]  Ming-Huwi Horng,et al.  The Construction of Support Vector Machine Classifier Using the Firefly Algorithm , 2015, Comput. Intell. Neurosci..

[2]  Shih-Wei Lin,et al.  Particle swarm optimization for parameter determination and feature selection of support vector machines , 2008, Expert Syst. Appl..

[3]  S. Ivvan Valdez,et al.  A Boltzmann based estimation of distribution algorithm , 2013 .

[4]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[5]  Xiaoli Zhang,et al.  An ACO-based algorithm for parameter optimization of support vector machines , 2010, Expert Syst. Appl..

[6]  Mingtian Zhou,et al.  Feature selection and parameter optimization for support vector machines: A new approach based on genetic algorithm with feature chromosomes , 2011, Expert Syst. Appl..

[7]  Heinz Mühlenbein,et al.  The Equation for Response to Selection and Its Use for Prediction , 1997, Evolutionary Computation.

[8]  Xin-She Yang,et al.  Bat algorithm: a novel approach for global engineering optimization , 2012, 1211.6663.

[9]  Shiliang Sun,et al.  A review of optimization methodologies in support vector machines , 2011, Neurocomputing.

[10]  Shigeo Abe.  Support Vector Machines for Pattern Classification , 2010, Advances in Pattern Recognition.

[11]  Dayou Liu,et al.  Evolving support vector machines using fruit fly optimization for medical data classification , 2016, Knowl. Based Syst..

[12]  Yunqiang Zhang,et al.  Machine training and parameter settings with social emotional optimization algorithm for support vector machine , 2015, Pattern Recognit. Lett..

[13]  Bernd Bischl,et al.  Effectiveness of Random Search in SVM hyper-parameter tuning , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[14]  Martín Carpio,et al.  Hyper-Parameter Tuning for Support Vector Machines by Estimation of Distribution Algorithms , 2017, Nature-Inspired Design of Hybrid Intelligent Systems.

[15]  Asdrúbal López Chau,et al.  Support vector machine classification for large datasets using decision tree and Fisher linear discriminant , 2014, Future Gener. Comput. Syst..

[16]  Mu-Chen Chen,et al.  Credit scoring with a data mining approach based on support vector machines , 2007, Expert Syst. Appl..

[17]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[18]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[19]  E. M. Wright,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[20]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[21]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[22]  Janez Brest,et al.  A Brief Review of Nature-Inspired Algorithms for Optimization , 2013, ArXiv.

[23]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[24]  Aditi Goel,et al.  Role of Kernel Parameters in Performance Evaluation of SVM , 2016, 2016 Second International Conference on Computational Intelligence & Communication Technology (CICT).

[25]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[26]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[27]  Zne-Jung Lee,et al.  Parameter determination of support vector machine and feature selection using simulated annealing approach , 2008, Appl. Soft Comput..