Feature selection with multi-objective genetic algorithm based on a hybrid filter and the symmetrical complementary coefficient

With the expansion of data size and data dimension, feature selection attracts more and more attention. In this paper, we propose a novel feature selection algorithm, namely, Hybrid filter and Symmetrical Complementary Coefficient based Multi-Objective Genetic Algorithm feature selection (HSMOGA). HSMOGA contains a new hybrid filter, Symmetrical Complementary Coefficient which is a well-performed metric of feature interactions proposed recently, and a novel way to limit feature subset’s size. A new Pareto-based ranking function is proposed when solving multi-objective problems. Besides, HSMOGA starts with a novel step called knowledge reserve, which precalculate the knowledge required for fitness function calculation and initial population generation. In this way, HSMOGA is classifier-independent in each generation, and its initial population generation makes full use of the knowledge of data set which makes solutions converge faster. Compared with other GA-based feature selection methods, HSMOGA has a much lower time complexity. According to experimental results, HSMOGA outperforms other nine state-of-art feature selection algorithms including five classic and four more recent algorithms in terms of kappa coefficient, accuracy, and G-mean for the data sets tested.

[1]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[2]  Max A. Little,et al.  Objective Automatic Assessment of Rehabilitative Speech Treatment in Parkinson's Disease , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[3]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[4]  David W. Coit,et al.  Multi-objective optimization using genetic algorithms: A tutorial , 2006, Reliab. Eng. Syst. Saf..

[5]  A. Ng.  Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[6]  Yong Zhang,et al.  Cost-sensitive feature selection using two-archive multi-objective artificial bee colony algorithm , 2019, Expert Syst. Appl..

[7]  Dunwei Gong,et al.  Binary differential evolution with self-learning for multi-objective feature selection , 2020, Inf. Sci..

[8]  Asit Kumar Das,et al.  Relevant feature selection and ensemble classifier design using bi-objective genetic algorithm , 2019, Knowledge and Information Systems.

[9]  Ram Sarkar,et al.  A wrapper-filter feature selection technique based on ant colony optimization , 2019, Neural Computing and Applications.

[10]  Yomi Kastro,et al.  Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks , 2019, Neural Computing and Applications.

[11]  Daniel Pizarro-Perez,et al.  Computer-Aided Classification of Gastrointestinal Lesions in Regular Colonoscopy , 2016, IEEE Transactions on Medical Imaging.

[12]  J. Clune,et al.  Scaling MAP-Elites to deep neuroevolution , 2020, GECCO.

[13]  Jaime S. Cardoso,et al.  Transfer Learning with Partial Observability Applied to Cervical Cancer Screening , 2017, IbPRIA.

[14]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Rui Zhang,et al.  Feature selection with Symmetrical Complementary Coefficient for quantifying feature interactions , 2019, Applied Intelligence.

[16]  Rui Zhang,et al.  A novel feature selection method considering feature interaction , 2015, Pattern Recognit..

[17]  Alberto Cano,et al.  Distributed Selection of Continuous Features in Multilabel Classification Using Mutual Information , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Tian Zheng,et al.  Interaction-based feature selection and classification for high-dimensional biological data , 2012, Bioinform..

[19]  Ping Zhang,et al.  Feature selection considering the composition of feature relevancy , 2018, Pattern Recognit. Lett..

[20]  Ron Kohavi,et al.  Irrelevant Features and the Subset Selection Problem , 1994, ICML.

[21]  Youwei Wang,et al.  Hybrid feature selection using component co-occurrence based feature relevance measurement , 2018, Expert Syst. Appl..

[22]  Hui-Huang Hsu,et al.  Hybrid feature selection by combining filters and wrappers , 2011, Expert Syst. Appl..

[23]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[24]  Chih-Cheng Hung,et al.  A Multi-objective hybrid filter-wrapper evolutionary approach for feature selection , 2018, Memetic Comput..

[25]  Jean-Baptiste Mouret,et al.  Illuminating search spaces by mapping elites , 2015, ArXiv.

[26]  Giovanni Iacca,et al.  Evaluating MAP-Elites on constrained optimization problems , 2019, GECCO.

[27]  Sanmay Das,et al.  Filters, Wrappers and a Boosting-Based Hybrid for Feature Selection , 2001, ICML.

[28]  Chih-Chieh Yang,et al.  Multiclass SVM-RFE for product form feature selection , 2008, Expert Syst. Appl..

[29]  Witold R. Rudnicki,et al.  Feature Selection with the Boruta Package , 2010 .

[30]  Luis González Abril,et al.  Ameva: An autonomous discretization algorithm , 2009, Expert Syst. Appl..

[31]  M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[32]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[33]  Hongmei He,et al.  Information gain directed genetic algorithm wrapper feature selection for credit rating , 2018, Appl. Soft Comput..

[34]  Azuraliza Abu Bakar,et al.  Hybrid feature selection based on enhanced genetic algorithm for text categorization , 2016, Expert Syst. Appl..

[35]  I-Cheng Yeh,et al.  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients , 2009, Expert Syst. Appl..

[36]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[37]  Jian Cheng,et al.  Multi-Objective Particle Swarm Optimization Approach for Cost-Based Feature Selection in Classification , 2017, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[38]  Sebastián Ventura,et al.  Distributed multi-label feature selection using individual mutual information measures , 2020, Knowl. Based Syst..

[39]  Youwei Wang,et al.  A new hybrid feature selection based on multi-filter weights and multi-feature weights , 2019, Applied Intelligence.

[40]  Xiaoyan Sun,et al.  Variable-Size Cooperative Coevolutionary Particle Swarm Optimization for Feature Selection on High-Dimensional Data , 2020, IEEE Transactions on Evolutionary Computation.

[41]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[42]  Ivan Bratko,et al.  Testing the significance of attribute interactions , 2004, ICML.

[43]  Federico Divina,et al.  Map-Elites Algorithm for Features Selection Problem , 2019, AMW.

[44]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[45]  Marek Lubicz,et al.  Boosted SVM for extracting rules from imbalanced data in application to prediction of the post-operative life expectancy in the lung cancer patients , 2014, Appl. Soft Comput..

[46]  Yuan-Shun Dai,et al.  Feature selection based on feature interactions with application to text categorization , 2019, Expert Syst. Appl..

[47]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[48]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[49]  Qinbao Song,et al.  Selecting Feature Subset via Constraint Association Rules , 2012, PAKDD.

[50]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[51]  Xiaoming Xu,et al.  A hybrid genetic algorithm for feature selection wrapper based on mutual information , 2007, Pattern Recognit. Lett..