Comparison of Embedded and Wrapper Approaches for Feature Selection in Support Vector Machines

Feature selection methods are generally divided into three categories: filter, wrapper and embedded approaches. In terms of learning performance, the filter approach is typically inferior compared to the other two because it does not use the target learning algorithm. The embedded and wrapper approaches are both considered high-performing. In this paper we compare the embedded and the wrapper approaches in the context of Support Vector Machines (SVMs). In the wrapper category, we compare well-known algorithms such as Genetic Algorithm (GA), Forward and Backward selection, and a new binary Particle Swarm Optimization (PSO) algorithm. For an embedded approach we devise a new heuristic algorithm based on Multiple Kernel Learning.

[1]  Xin Yao,et al.  A Survey on Evolutionary Computation Approaches to Feature Selection , 2016, IEEE Transactions on Evolutionary Computation.

[2]  Ling Wang,et al.  A Novel PSO-Inspired Probability-based Binary Optimization Algorithm , 2008, 2008 International Symposium on Information Science and Engineering.

[3]  Zbigniew Michalewicz,et al.  Stability Analysis of the Particle Swarm Optimization Without Stagnation Assumption , 2016, IEEE Transactions on Evolutionary Computation.

[4]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[5]  Pedro Larrañaga,et al.  A review of feature selection techniques in bioinformatics , 2007, Bioinform..

[6]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[7]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[8]  Mengjie Zhang,et al.  A New Representation in PSO for Discretisation-Based Feature Selection , 2017 .

[9]  Mengjie Zhang,et al.  Particle Swarm Optimisation for Feature Selection in Classification , 2014 .

[10]  Mengjie Zhang,et al.  Particle Swarm Optimization for Feature Selection in Classification: A Multi-Objective Approach , 2013, IEEE Transactions on Cybernetics.

[11]  M. Kloft,et al.  l p -Norm Multiple Kernel Learning , 2011 .

[12]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[13]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[14]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[15]  Kourosh Neshatian,et al.  Multiple Kernel Learning with One-Level Optimization of Radius and Margin , 2017, Australasian Conference on Artificial Intelligence.