Hybridization of feature selection and feature weighting for high dimensional data

The classification of high dimensional data is a challenging problem due to the presence of redundant and irrelevant features in a higher amount. These unwanted features degrade accuracy and increase the computational complexity of machine learning algorithms. In this paper, we propose a hybrid method that integrates the complementary strengths of feature selection and feature weighting approaches for improving the classification of high dimensional data on the Nearest Neighbor classifier. Specifically, we suggest four strategies that combine filter and wrapper methods of feature selection and feature weighting. Experiments are performed on 12 high dimensional datasets and outcomes are supported by Friedman as well as Holm statistical tests for validation. Extended Adjusted Ratio of Ratios is used to recognize the best method considering accuracy, feature selection, and runtime. The results show that two proposed strategies outperform other well-known methods in accuracy and features reduction. The hybrid feature selection-feature weighting wrapper method is the best among all in accuracy while the hybrid feature selection filter-feature weighting wrapper method is the most suitable for reducing features and runtime. Thus, the promising outcomes validate the importance of hybridizing feature selection and feature weighting while dealing with high dimensional data.

[1]  Francisco Herrera,et al.  A survey on data preprocessing for data stream mining: Current status and future directions , 2017, Neurocomputing.

[2]  Gholamreza Haffari,et al.  Data-dependent dissimilarity measure: an effective alternative to geometric distance measures , 2017, Knowledge and Information Systems.

[3]  Mengjie Zhang,et al.  A binary ABC algorithm based on advanced similarity scheme for feature selection , 2015, Appl. Soft Comput..

[4]  Naftali Tishby,et al.  Margin based feature selection - theory and algorithms , 2004, ICML.

[5]  Yuming Zhou,et al.  A Feature Subset Selection Algorithm Automatic Recommendation Method , 2013, J. Artif. Intell. Res..

[6]  Hongzhi Wang,et al.  Novel fruit fly optimization algorithm with trend search and co-evolution , 2018, Knowl. Based Syst..

[7]  Ibrahim Berkan Aydilek.  A hybrid firefly and particle swarm optimization algorithm for computationally expensive numerical problems , 2018, Appl. Soft Comput..

[8]  José Cristóbal Riquelme Santos,et al.  An evolutionary voting for k-nearest neighbours , 2016, Expert Syst. Appl..

[9]  Sinisa Todorovic,et al.  Local-Learning-Based Feature Selection for High-Dimensional Data Analysis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  B. S. Harish,et al.  A New Feature Selection Method based on Intuitionistic Fuzzy Entropy to Categorize Text Documents , 2018, Int. J. Interact. Multim. Artif. Intell..

[11]  Hao Dong,et al.  An improved particle swarm optimization for feature selection , 2011 .

[12]  Kewei Cheng,et al.  Feature Selection , 2016, ACM Comput. Surv..

[13]  Aboul Ella Hassanien,et al.  Chaotic antlion algorithm for parameter optimization of support vector machine , 2018, Applied Intelligence.

[14]  Parham Moradi,et al.  OSFSMI: Online stream feature selection method based on mutual information , 2017, Appl. Soft Comput..

[15]  Michael K. Ng,et al.  Automated variable weighting in k-means type clustering , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  M. Hariharan,et al.  Sine–cosine algorithm for feature selection with elitism strategy and new updating mechanism , 2017, Neural Comput. Appl..

[17]  Satvir Singh,et al.  An Effective Hybrid Butterfly Optimization Algorithm with Artificial Bee Colony for Numerical Optimization , 2017, Int. J. Interact. Multim. Artif. Intell..

[18]  Hao Liao,et al.  An efficient semi-supervised representatives feature selection algorithm based on information theory , 2017, Pattern Recognit..

[19]  Jerome H. Friedman,et al.  Recent Advances in Predictive (Machine) Learning , 2006, J. Classif..

[20]  Daphne Koller,et al.  Toward Optimal Feature Selection , 1996, ICML.

[21]  Yijun Sun,et al.  Iterative RELIEF for Feature Weighting: Algorithms, Theories, and Applications , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Hao Wang,et al.  Online Feature Selection with Streaming Features , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  G. F. Hughes,et al.  On the mean accuracy of statistical pattern recognizers , 1968, IEEE Trans. Inf. Theory.

[24]  Xindong Wu,et al.  LOFS: Library of Online Streaming Feature Selection , 2016, Knowl. Based Syst..

[25]  Lawrence Davis,et al.  A Hybrid Genetic Algorithm for Classification , 1991, IJCAI.

[26]  Seyed Mohammad Mirjalili,et al.  The Ant Lion Optimizer , 2015, Adv. Eng. Softw..

[27]  Mark A. Hall,et al.  Correlation-based Feature Selection for Machine Learning , 2003 .

[28]  Sayan Mukherjee,et al.  Feature Selection for SVMs , 2000, NIPS.

[29]  Dimitrios Gunopulos,et al.  Locally adaptive metrics for clustering high dimensional data , 2007, Data Mining and Knowledge Discovery.

[30]  Aboul Ella Hassanien,et al.  Sine cosine optimization algorithm for feature selection , 2016, 2016 International Symposium on INnovations in Intelligent SysTems and Applications (INISTA).

[31]  Songyot Nakariyakul,et al.  High-dimensional hybrid feature selection using interaction information-guided search , 2018, Knowl. Based Syst..

[32]  Mohamed Medhat Gaber,et al.  Advances in data stream mining , 2012, WIREs Data Mining Knowl. Discov..

[33]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[34]  M. Narasimha Murty,et al.  Classification by majority voting in feature partitions , 2016, Int. J. Inf. Decis. Sci..

[35]  J. Carroll,et al.  Synthesized clustering: A method for amalgamating alternative clustering bases with differential weighting of variables , 1984 .

[36]  Honglun Wang,et al.  Dynamic Adaptive Ant Lion Optimizer applied to route planning for unmanned aerial vehicle , 2017, Soft Comput..

[37]  Yaochu Jin,et al.  Feature selection for high-dimensional classification using a competitive swarm optimizer , 2016, Soft Computing.

[38]  Yogesh R. Shepal.  A Fast Clustering-Based Feature Subset Selection Algorithm for High Dimensional Data , 2014 .

[39]  Francisco Herrera,et al.  Enhancing evolutionary instance selection algorithms by means of fuzzy rough set based feature selection , 2012, Inf. Sci..

[40]  Zhaohong Deng,et al.  Robust Relief-Feature Weighting, Margin Maximization, and Fuzzy Optimization , 2010, IEEE Transactions on Fuzzy Systems.

[41]  Hao Liao,et al.  Unsupervised feature selection based on Markov blanket and particle swarm optimization , 2017 .

[42]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[43]  Javier Pérez-Rodríguez,et al.  Simultaneous instance and feature selection and weighting using evolutionary computation: Proposal and study , 2015, Appl. Soft Comput..

[44]  Francisco Herrera,et al.  Integrating Instance Selection, Instance Weighting, and Feature Weighting for Nearest Neighbor Classifiers by Coevolutionary Algorithms , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[45]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[46]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[47]  Enrique Vidal,et al.  A class-dependent weighted dissimilarity measure for nearest neighbor classification problems , 2000, Pattern Recognit. Lett..

[48]  George D. C. Cavalcanti,et al.  Combining global optimization algorithms with a simple adaptive distance for feature selection and weighting , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[49]  Enrique Alba,et al.  Two hybrid wrapper-filter feature selection algorithms applied to high-dimensional microarray experiments , 2016, Appl. Soft Comput..

[50]  Ping Zhang,et al.  Class-specific mutual information variation for feature selection , 2018, Pattern Recognit..

[51]  Michael K. Ng,et al.  An Entropy Weighting k-Means Algorithm for Subspace Clustering of High-Dimensional Sparse Data , 2007, IEEE Transactions on Knowledge and Data Engineering.

[52]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[53]  Rubén González Crespo,et al.  MOVPSO: Vortex Multi-Objective Particle Swarm Optimization , 2017, Appl. Soft Comput..

[54]  Isaac Meilijson,et al.  Splitting matters: how monotone transformation of predictor variables may improve the predictions of decision tree models , 2016, ArXiv.

[55]  Anil K. Jain,et al.  Dimensionality reduction using genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[56]  Andries Petrus Engelbrecht,et al.  A Cooperative approach to particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[57]  Li-Yeh Chuang,et al.  A hybrid feature selection method for DNA microarray data , 2011, Comput. Biol. Medicine.

[58]  Yunming Ye,et al.  A feature group weighting method for subspace clustering of high-dimensional data , 2012, Pattern Recognit..

[59]  Huan Liu,et al.  Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution , 2003, ICML.

[60]  Hossein Nezamabadi-pour,et al.  Ensemble of Filter-Based Rankers to Guide an Epsilon-Greedy Swarm Optimizer for High-Dimensional Feature Subset Selection , 2017, Inf..

[61]  Ahmed Bouridane,et al.  Simultaneous feature selection and feature weighting using Hybrid Tabu Search/K-nearest neighbor classifier , 2007, Pattern Recognit. Lett..

[62]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[63]  Vimal Kumar Dubey,et al.  A cluster-filter feature selection approach , 2016, 2016 International Conference on ICT in Business Industry & Government (ICTBIG).

[64]  Akash Saxena,et al.  Performance Evaluation of Antlion Optimizer Based Regulator in Automatic Generation Control of Interconnected Power System , 2016 .

[65]  Yuming Zhou,et al.  Selecting feature subset for high dimensional data via the propositional FOIL rules , 2013, Pattern Recognit..

[66]  Jon Atli Benediktsson,et al.  Feature Selection Based on Hybridization of Genetic Algorithm and Particle Swarm Optimization , 2015, IEEE Geoscience and Remote Sensing Letters.

[67]  Nicolás García-Pedrajas,et al.  Random feature weights for decision tree ensemble construction , 2012, Inf. Fusion.