A multi-objective genetic algorithm for simultaneous model and feature selection for support vector machines

The Support Vector Machines (SVM) constitute a very powerful technique for pattern classification problems. However, its efficiency in practice depends highly on the selection of the kernel function type and relevant parameter values. Selecting relevant features is another factor that can also impact the performance of SVM. The identification of the best set of parameters values for a classification model such as SVM is considered as an optimization problem. Thus, in this paper, we aim to simultaneously optimize SVMs parameters and feature subset using different kernel functions. We cast this problem as a multi-objective optimization problem, where the classification accuracy, the number of support vectors, the margin and the number of selected features define our objective functions. To solve this optimization problem, a method based on multi-objective genetic algorithm NSGA-II is suggested. A multi-criteria selection operator for our NSGA-II is also introduced. The proposed method is tested on some benchmark data-sets. The experimental results show the efficiency of the proposed method where features were reduced and the classification accuracy has been improved.

[1]  Kalyanmoy Deb,et al.  Omni-optimizer: A generic evolutionary algorithm for single and multi-objective optimization , 2008, Eur. J. Oper. Res..

[2]  Anil K. Jain,et al.  Dimensionality reduction using genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[3]  Masoud Nikravesh,et al.  Feature Extraction - Foundations and Applications , 2006, Feature Extraction.

[4]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[5]  Farid Melgani,et al.  Classification of Electrocardiogram Signals With Support Vector Machines and Particle Swarm Optimization , 2008, IEEE Transactions on Information Technology in Biomedicine.

[6]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[7]  Guoqiang Peter Zhang,et al.  Neural networks for classification: a survey , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[8]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[9]  Wei-Chun Kao,et al.  Radius Margin Bounds for Support Vector . . . , 2003 .

[10]  Zhi Chen,et al.  A Parallel Genetic Algorithm Based Feature Selection and Parameter Optimization for Support Vector Machine , 2016, Sci. Program..

[11]  Bernhard Schölkopf,et al.  Feature selection for support vector machines by means of genetic algorithm , 2003, Proceedings. 15th IEEE International Conference on Tools with Artificial Intelligence.

[12]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[13]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[14]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[15]  Kalyanmoy Deb,et al.  Self-Adaptive Genetic Algorithms with Simulated Binary Crossover , 2001, Evolutionary Computation.

[16]  Xing Li,et al.  Evolving support vector machine parameters , 2002, Proceedings. International Conference on Machine Learning and Cybernetics.

[17]  J. Rajapakse,et al.  Gene Classification Using Codon Usage and Support Vector Machines , 2009, IEEE/ACM Transactions on Computational Biology & Bioinformatics.

[18]  Yao-Nan Wang,et al.  A method to choose kernel function and its parameters for support vector machines , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[19]  S. Balakrishnan,et al.  Feature Selection Using FCBF in Type II Diabetes Databases , 2009 .

[20]  Mingtian Zhou,et al.  Feature selection and parameter optimization for support vector machines: A new approach based on genetic algorithm with feature chromosomes , 2011, Expert Syst. Appl..

[21]  Zhang Hongtao,et al.  Chaos Optimization Method of SVM Parameters Selection for Chaotic Time Series Forecasting , 2012 .

[22]  Fong-Ching Yuan.  Parameters Optimization Using Genetic Algorithms in Support Vector Regression for Sales Volume Forecasting , 2012 .

[23]  Shih-Wei Lin,et al.  Particle swarm optimization for parameter determination and feature selection of support vector machines , 2008, Expert Syst. Appl..

[24]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[25]  Yi-Hung Huang,et al.  Feature Selection and Parameter Optimization of Support Vector Machines Based on Modified Cat Swarm Optimization , 2015, Int. J. Distributed Sens. Networks.

[26]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[27]  Ming-Chi Lee,et al.  Using support vector machine with a hybrid feature selection method to the stock trend prediction , 2009, Expert Syst. Appl..

[28]  Malek Sarhani,et al.  Feature selection and parameter optimization of support vector regression for electric load forecasting , 2016, 2016 International Conference on Electrical and Information Technologies (ICEIT).

[29]  Kunlun Li,et al.  Parameter Optimization for Support Vector Machine Based on Nested Genetic Algorithms , 2015 .

[30]  Aixia Guo,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2014 .

[31]  Delmiro Fernandez-Reyes,et al.  Adapting multiple kernel parameters for support vector machines using genetic algorithms , 2005, 2005 IEEE Congress on Evolutionary Computation.

[32]  Huanhuan Chen,et al.  Evolving Least Squares Support Vector Machines for Stock Market Trend Mining , 2009, IEEE Trans. Evol. Comput..

[33]  Andy,et al.  Optimization Features Using GA-SVM Approach , 2015 .

[34]  Christian Dipl.-Ing. Walther,et al.  Parameter optimization for support vector machines by using a multicriteria genetic algorithm for classification of sleep-stages , 2010 .

[35]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[36]  Marco Sciandrone,et al.  Please Scroll down for Article Optimization Methods and Software Feature Selection Combining Linear Support Vector Machines and Concave Optimization Feature Selection Combining Linear Support Vector Machines and Concave Optimization , 2022 .

[37]  Kezhi Mao,et al.  Feature subset selection for support vector machines through discriminative function pruning analysis , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[38]  Chih-Hung Wu,et al.  A Novel hybrid genetic algorithm for kernel function and parameter optimization in support vector regression , 2009, Expert Syst. Appl..

[39]  Cheng-Lung Huang,et al.  A GA-based feature selection and parameters optimizationfor support vector machines , 2006, Expert Syst. Appl..

[40]  Ashutosh,et al.  Evolutionary Selection of Kernels in Support Vector Machines , 2006, 2006 International Conference on Advanced Computing and Communications.

[41]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[42]  Shuang Liu,et al.  A new weighted support vector machine with GA-based parameter selection , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[43]  Liu Fang,et al.  Choosing multiple parameters for SVM based on genetic algorithm , 2002, 6th International Conference on Signal Processing, 2002..

[44]  Xin Yao,et al.  A Survey on Evolutionary Computation Approaches to Feature Selection , 2016, IEEE Transactions on Evolutionary Computation.

[45]  M. Pardo,et al.  Classification of electronic nose data with support vector machines , 2005 .