Feature weighting and SVM parameters optimization based on genetic algorithms for classification problems

Support Vector Machines (SVMs) are widely known as an efficient supervised learning model for classification problems. However, the success of an SVM classifier depends on the perfect choice of its parameters as well as the structure of the data. Thus, the aim of this research is to simultaneously optimize the parameters and feature weighting in order to increase the strength of SVMs. We propose a novel hybrid model, the combination of genetic algorithms (GAs) and SVMs, for feature weighting and parameter optimization to solve classification problems efficiently. We call it as the GA-SVM model. Our GA is designed with a special direction-based crossover operator. Experiments were conducted on several real-world datasets using the proposed model and Grid Search, a traditional method of searching optimal parameters. The results show that the GA-SVM model achieves significant improvement in the performance of classification on all the datasets in comparison with Grid Search. In terms of accuracy, out method is competitive with some state-of-the-art techniques for feature selection and feature weighting.

[1]  Jing Wen,et al.  Text Categorization System for Stock Prediction , 2015 .

[2]  Ingoo Han,et al.  Hybrid genetic algorithms and support vector machines for bankruptcy prediction , 2006, Expert Syst. Appl..

[3]  Dimitrios Gunopulos,et al.  Locally Adaptive Metric Nearest-Neighbor Classification , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Ahmed Bouridane,et al.  Simultaneous feature selection and feature weighting using Hybrid Tabu Search/K-nearest neighbor classifier , 2007, Pattern Recognit. Lett..

[5]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[6]  Nagamma Patil,et al.  Genetic algorithm based wrapper feature selection on hybrid prediction model for analysis of high dimensional data , 2014, 2014 9th International Conference on Industrial and Information Systems (ICIIS).

[7]  Guoyin Wang,et al.  Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing , 2013, Lecture Notes in Computer Science.

[8]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[9]  Harris Wu,et al.  The effects of fitness functions on genetic programming-based ranking discovery forWeb search , 2004, J. Assoc. Inf. Sci. Technol..

[10]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[11]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[12]  David E. Goldberg,et al.  Genetic algorithms and Machine Learning , 1988, Machine Learning.

[13]  Edison A. Roxas,et al.  Recognition of handwritten alphanumeric characters using Projection Histogram and Support Vector Machine , 2015, 2015 International Conference on Humanoid, Nanotechnology, Information Technology,Communication and Control, Environment and Management (HNICEM).

[14]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[15]  Steven M. LaValle,et al.  On the Relationship between Classical Grid Search and Probabilistic Roadmaps , 2004, Int. J. Robotics Res..

[16]  Bernhard Schölkopf,et al.  Feature selection for support vector machines by means of genetic algorithm , 2003, Proceedings. 15th IEEE International Conference on Tools with Artificial Intelligence.

[17]  Malcolm Sambridge,et al.  Genetic algorithms: a powerful tool for large-scale nonlinear optimization problems , 1994 .

[18]  Gustavo E. A. P. A. Batista,et al.  An analysis of four missing data treatment methods for supervised learning , 2003, Appl. Artif. Intell..

[19]  Tuba Yavuz.  Application of k Nearest Neighbor on Feature Projections Classi er to Text Categorization , 2004 .

[20]  Chang-Hwan Lee.  A gradient approach for value weighted classification learning in naive Bayes , 2015, Knowl. Based Syst..

[21]  Chih-Hung Wu,et al.  A real-valued genetic algorithm to optimize the parameters of support vector machine for predicting bankruptcy , 2007, Expert Syst. Appl..

[22]  Shasha Wang,et al.  Deep feature weighting for naive Bayes and its application to text classification , 2016, Eng. Appl. Artif. Intell..

[23]  Francisco Herrera,et al.  Statistical computation of feature weighting schemes through data estimation for nearest neighbor classifiers , 2014, Pattern Recognit..

[24]  David G. Lowe,et al.  Similarity Metric Learning for a Variable-Kernel Classifier , 1995, Neural Computation.

[25]  H. Altay Güvenir,et al.  WEIGHTED K NEAREST NEIGHBOR CLASSIFICATION ON FEATURE PROJECTIONS , 2010 .

[26]  J. M. DeLeo,et al.  Essential roles for receiver operating characteristic (ROC) methodology in classifier neural network applications , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[27]  Enrique Vidal,et al.  A class-dependent weighted dissimilarity measure for nearest neighbor classification problems , 2000, Pattern Recognit. Lett..

[28]  Chengqi Zhang,et al.  Self-adaptive attribute weighting for Naive Bayes classification , 2015, Expert Syst. Appl..

[29]  Cheng-Lung Huang,et al.  A GA-based feature selection and parameters optimizationfor support vector machines , 2006, Expert Syst. Appl..

[30]  Nitesh V. Chawla,et al.  Editorial: special issue on learning from imbalanced data sets , 2004, SKDD.

[31]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[32]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[33]  Richard J. Enbody,et al.  Further Research on Feature Selection and Classification Using Genetic Algorithms , 1993, ICGA.

[34]  Dae-Ki Kang,et al.  Experimental analysis of naïve Bayes classifier based on an attribute weighting framework with smooth kernel density estimations , 2015, Applied Intelligence.

[35]  ZhangPeng,et al.  Self-adaptive attribute weighting for Naive Bayes classification , 2015 .

[36]  Gill Bejerano,et al.  Automata Learning and Stochastic Modeling for Biosequence Analysis , 2003 .

[37]  Vaishali Ganganwar,et al.  An overview of classification algorithms for imbalanced datasets , 2012 .

[38]  Anil K. Jain,et al.  Dimensionality reduction using genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[39]  Giles M. Foody,et al.  The effect of mis-labeled training data on the accuracy of supervised image classification by SVM , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[40]  Jerzy W. Grzymala-Busse,et al.  Handling Missing Attribute Values in Preterm Birth Data Sets , 2005, RSFDGrC.

[41]  Bingru Yang,et al.  A SVM Regression Based Approach to Filling in Missing Values , 2005, KES.

[42]  Harris Wu,et al.  The effects of fitness functions on genetic programming-based ranking discovery for Web search: Research Articles , 2004 .

[43]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[44]  Ajalmar R. da Rocha Neto,et al.  Novel approaches using evolutionary computation for sparse least square support vector machines , 2015, Neurocomputing.

[45]  Lawrence Davis,et al.  A Hybrid Genetic Algorithm for Classification , 1991, IJCAI.