Differential evolution for feature selection: a fuzzy wrapper–filter approach

The selection of an optimal feature subset from all available features in the data is a vital task of data pre-processing used for several purposes such as the dimensionality reduction, the computational complexity reduction required for data processing (e.g., clustering, classification and regression) and the performance enhancement of a data processing technique. To serve such purposes, feature selection approaches which are fundamentally categorized into filters and wrappers try to eliminate irrelevant, redundant and erroneous features in the data. Each category comes with its own advantages and disadvantages. While wrappers can generally provide higher classification performance than filters, filters are computationally more efficient than wrappers. In order to bring the advantages of wrappers and filters together, i.e., to get higher classification performance with smaller feature subset size in a shorter time, this paper proposes a differential evolution approach combining filter and wrapper approaches through an improved information theoretic local search mechanism which is based on the concepts of fuzziness to cope with both continuous and discrete datasets. To show the superiority of the proposed approach, it is examined and compared with traditional and recent evolutionary feature selection approaches on several benchmarks from different well-known data repositories.

[1]  Hao Dong,et al.  An improved particle swarm optimization for feature selection , 2011 .

[2]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[3]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[4]  Xin Yao,et al.  A Survey on Evolutionary Computation Approaches to Feature Selection , 2016, IEEE Transactions on Evolutionary Computation.

[5]  Sung-Bae Cho,et al.  Efficient huge-scale feature selection with speciated genetic algorithm , 2005 .

[6]  Mengjie Zhang,et al.  Differential evolution for filter feature selection based on information theory and feature ranking , 2018, Knowl. Based Syst..

[7]  Mengjie Zhang,et al.  Particle swarm optimisation for feature selection in classification: Novel initialisation and updating mechanisms , 2014, Appl. Soft Comput..

[8]  Mengjie Zhang,et al.  Pareto front feature selection based on artificial bee colony optimization , 2018, Inf. Sci..

[9]  Alina A. von Davier,et al.  Cross-Validation , 2014 .

[10]  Mostafa A. El-Hosseini,et al.  Design of optimal PID controller using hybrid differential evolution and particle swarm optimization with an aging leader and challengers , 2016, Appl. Soft Comput..

[11]  Li-Yeh Chuang,et al.  A Hybrid Both Filter and Wrapper Feature Selection Method for Microarray Classification , 2016, ArXiv.

[12]  Samaher AlJanabi,et al.  Assessing the suitability of soft computing approaches for forest fires prediction , 2018, Applied Computing and Informatics.

[13]  Cheng-Lung Huang,et al.  A distributed PSO-SVM hybrid system with feature selection and parameter optimization , 2008, Appl. Soft Comput..

[14]  Gamini Dissanayake,et al.  Driver Drowsiness Classification Using Fuzzy Wavelet-Packet-Based Feature-Extraction Algorithm , 2011, IEEE Transactions on Biomedical Engineering.

[15]  Mengjie Zhang,et al.  A differential evolution based feature selection approach using an improved filter criterion , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[16]  Ian Witten,et al.  Data Mining , 2000 .

[17]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Jintao Zhang,et al.  The BioAssay network and its implications to future therapeutic discovery , 2011, BMC Bioinformatics.

[19]  Lale Özbakir,et al.  A cost-sensitive classification algorithm: BEE-Miner , 2016, Knowl. Based Syst..

[20]  Y.-C. Lee,et al.  Feature selection and classification by using grid computing based evolutionary approach for the microarray data , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[21]  Samaher Hussein Ali,et al.  A novel tool (FP-KC) for handle the three main dimensions reduction and association rule mining , 2012, 2012 6th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications (SETIT).

[22]  Bin Wei,et al.  Comparison between differential evolution and particle swarm optimization algorithms , 2014, 2014 IEEE International Conference on Mechatronics and Automation.

[23]  Mengjie Zhang,et al.  Binary PSO and Rough Set Theory for Feature Selection: a Multi-objective filter Based Approach , 2014, Int. J. Comput. Intell. Appl..

[24]  Zehong Yang,et al.  A novel hybrid feature selection algorithm: using ReliefF estimation for GA-Wrapper search , 2003, Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.03EX693).

[25]  Jacek M. Zurada,et al.  Normalized Mutual Information Feature Selection , 2009, IEEE Transactions on Neural Networks.

[26]  Thomas G. Dietterich,et al.  Learning Boolean Concepts in the Presence of Many Irrelevant Features , 1994, Artif. Intell..

[27]  Xindong Wu,et al.  Gene expression analyses using Genetic Algorithm based hybrid approaches , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[28]  B. V. Babu,et al.  Differential evolution strategies for optimal design of shell-and-tube heat exchangers , 2007 .

[29]  A. Wayne Whitney,et al.  A Direct Method of Nonparametric Measurement Selection , 1971, IEEE Transactions on Computers.

[30]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[31]  Ahmed Al-Ani.  Ant Colony Optimization for Feature Subset Selection , 2005, WEC.

[32]  Mengjie Zhang,et al.  Particle swarm optimisation for feature selection: A hybrid filter-wrapper approach , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[33]  Shyam Visweswaran,et al.  Application of an efficient Bayesian discretization method to biomedical data , 2011, BMC Bioinformatics.

[34]  Lale Özbakir,et al.  Cost-sensitive meta-learning classifiers: MEPAR-miner and DIFACONN-miner , 2016, Knowl. Based Syst..

[35]  Thomas Marill,et al.  On the effectiveness of receptors in recognition systems , 1963, IEEE Trans. Inf. Theory.

[36]  Deng Cai,et al.  Laplacian Score for Feature Selection , 2005, NIPS.

[37]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[38]  Mengjie Zhang,et al.  A binary ABC algorithm based on advanced similarity scheme for feature selection , 2015, Appl. Soft Comput..

[39]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[40]  Rich Caruana,et al.  Greedy Attribute Selection , 1994, ICML.

[41]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[42]  Fernando José Von Zuben,et al.  Multi-objective feature selection using a Bayesian artificial immune system , 2010, Int. J. Intell. Comput. Cybern..

[43]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[44]  Shih-Wei Lin,et al.  Particle swarm optimization for parameter determination and feature selection of support vector machines , 2008, Expert Syst. Appl..

[45]  Mengjie Zhang,et al.  A Dimension Reduction Approach to Classification Based on Particle Swarm Optimisation and Rough Set Theory , 2012, Australasian Conference on Artificial Intelligence.

[46]  Mengjie Zhang,et al.  An archive based particle swarm optimisation for feature selection in classification , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[47]  Eibe Frank,et al.  Large-scale attribute selection using wrappers , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[48]  Adel Al-Jumaily,et al.  A Combined Ant Colony and Differential Evolution Feature Selection Algorithm , 2008, ANTS Conference.

[49]  Byung Ro Moon,et al.  Hybrid Genetic Algorithms for Feature Selection , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Enrique Alba,et al.  Two hybrid wrapper-filter feature selection algorithms applied to high-dimensional microarray experiments , 2016, Appl. Soft Comput..

[51]  Abdul Rahman Ramli,et al.  Feature selection for high dimensional data: An evolutionary filter approach. , 2011 .

[52]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[53]  Samaher Al-Janabi,et al.  Pragmatic Miner to Risk Analysis for Intrusion Detection (PMRA-ID) , 2017, SCDS.

[54]  Sébastien Paris,et al.  Application of global optimization methods to model and feature selection , 2012, Pattern Recognit..

[55]  Andreas König,et al.  Feature-Level Fusion by Multi-Objective Binary Particle Swarm Based Unbiased Feature Selection for Optimized Sensor System Design , 2006, 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[56]  Mengjie Zhang,et al.  A multi-objective particle swarm optimisation for filter-based feature selection in classification problems , 2012, Connect. Sci..

[57]  John C. Kelly,et al.  International Journal of Intelligent Computing and Cybernetics Genetic and evolutionary biometrics : Exploring value preference space for hybrid feature weighting and selection , 2016 .

[58]  Xin Yao,et al.  Feature Selection for Microarray Data Using Least Squares SVM and Particle Swarm Optimization , 2005, 2005 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology.

[59]  Samaher Al-Janabi,et al.  Soft Mathematical System to Solve Black Box Problem through Development the FARB Based on Hyperbolic and Polynomial Functions , 2017, 2017 10th International Conference on Developments in eSystems Engineering (DeSE).

[60]  Mengjie Zhang,et al.  Improving feature ranking for biomarker discovery in proteomics mass spectrometry data using genetic programming , 2014, Connect. Sci..

[61]  Rami N. Khushaba,et al.  Feature subset selection using differential evolution and a wheel based search strategy , 2013, Swarm Evol. Comput..

[62]  Zexuan Zhu,et al.  Wrapper–Filter Feature Selection Algorithm Using a Memetic Framework , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[63]  Mengjie Zhang,et al.  Novel Initialisation and Updating Mechanisms in PSO for Feature Selection in Classification , 2013, EvoApplications.

[64]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[65]  Yan Dong,et al.  Feature Selection with Discrete Binary Differential Evolution , 2009, 2009 International Conference on Artificial Intelligence and Computational Intelligence.

[66]  Mengjie Zhang,et al.  Particle Swarm Optimisation and Statistical Clustering for Feature Selection , 2013, Australasian Conference on Artificial Intelligence.

[67]  Myong Kee Jeong,et al.  An evolutionary algorithm with the partial sequential forward floating search mutation for large-scale feature selection problems , 2015, J. Oper. Res. Soc..

[68]  Xiaoming Xu,et al.  A hybrid genetic algorithm for feature selection wrapper based on mutual information , 2007, Pattern Recognit. Lett..

[69]  Bhaskar Gupta,et al.  Performance Comparison of Differential Evolution, Particle Swarm Optimization and Genetic Algorithm in the Design of Circularly Polarized Microstrip Antennas , 2014, IEEE Transactions on Antennas and Propagation.

[70]  Hong Hu,et al.  Using PSO algorithm to evolve an optimum input subset for a SVM in time series forecasting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[71]  Barnali Sahu,et al.  A Novel Feature Selection Algorithm using Particle Swarm Optimization for Cancer Microarray Data , 2012 .

[72]  Amir-Massoud Bidgoli,et al.  A Hybrid Feature Selection Method to Improve Performance of a Group of Classification Algorithms , 2013, ArXiv.

[73]  Nikhil R. Pal,et al.  Genetic programming for simultaneous feature selection and classifier design , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[74]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[75]  I. Jolliffe.  Principal Component Analysis , 2002 .

[76]  Mengjie Zhang,et al.  Filter based backward elimination in wrapper based PSO for feature selection in classification , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[77]  Jens Myrup Pedersen,et al.  A novel methodology towards a trusted environment in mashup web applications , 2015, Comput. Secur..

[78]  C. C. O. Ramos,et al.  New Insights on Nontechnical Losses Characterization Through Evolutionary-Based Feature Selection , 2012, IEEE Transactions on Power Delivery.

[79]  Ramin Mansouri,et al.  Optimization of the Water Distribution Networks with Differential Evolution (DE) and Mixed Integer Linear Programming (MILP) , 2015 .