Feature selection based on quality of information

Feature selection as one of the key problems of data preprocessing is a hot research topic in pattern recognition, machine learning, and data mining. Evaluating the relevance between features based on information theory is a popular and effective method. However, very little research pays attention to the distinguishing ability of feature, i.e., the degree of a feature distinguishes a given sample with other samples. In this paper, we propose a new feature selection method based on the distinguishing ability of feature. First, we define the concept of maximum-nearest-neighbor, and use this concept to discriminate the nearest neighbors of samples. Then, we present a new measure method for evaluating the quality of feature. Finally, the proposed algorithm is tested on benchmark datasets. Experimental results show that the proposed algorithm can effectively select a discriminative feature subset, and performs as well as or better than other popular feature selection algorithms.

[1]  Xin Yao,et al.  DDD: A New Ensemble Approach for Dealing with Concept Drift , 2012, IEEE Transactions on Knowledge and Data Engineering.

[2]  Marcel J. T. Reinders,et al.  Random subspace method for multivariate feature selection , 2006, Pattern Recognit. Lett..

[3]  Lei Wang,et al.  On Similarity Preserving Feature Selection , 2013, IEEE Transactions on Knowledge and Data Engineering.

[4]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[5]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[6]  Dahua Lin,et al.  Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion , 2006, ECCV.

[7]  Jugal K. Kalita,et al.  MIFS-ND: A mutual information-based feature selection method , 2014, Expert Syst. Appl..

[8]  Qinghua Hu,et al.  Rank Entropy-Based Decision Trees for Monotonic Classification , 2012, IEEE Transactions on Knowledge and Data Engineering.

[9]  Bart Kosko,et al.  Fuzzy entropy and conditioning , 1986, Inf. Sci..

[10]  A GámezJosé,et al.  Speeding up incremental wrapper feature subset selection with Naive Bayes classifier , 2014 .

[11]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[12]  Hua Zhao,et al.  Mixed feature selection in incomplete decision table , 2014, Knowl. Based Syst..

[13]  Witold Pedrycz,et al.  Measuring relevance between discrete and continuous features based on neighborhood mutual information , 2011, Expert Syst. Appl..

[14]  Pei Zhang,et al.  Classifying Categorical Data by Rule-Based Neighbors , 2011, 2011 IEEE 11th International Conference on Data Mining.

[15]  Tommy W. S. Chow,et al.  Heterogeneous feature subset selection using mutual information-based feature transformation , 2015, Neurocomputing.

[16]  Gavin Brown,et al.  Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection , 2012, J. Mach. Learn. Res..

[17]  Larry A. Rendell,et al.  The Feature Selection Problem: Traditional Methods and a New Algorithm , 1992, AAAI.

[18]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[19]  Wangmeng Zuo,et al.  Consistency analysis on orientation features for fast and accurate palmprint identification , 2014, Inf. Sci..

[20]  HuQinghua,et al.  Multi-label feature selection based on max-dependency and min-redundancy , 2015 .

[21]  Huan Liu,et al.  Spectral feature selection for supervised and unsupervised learning , 2007, ICML '07.

[22]  James Bailey,et al.  Effective global approaches for mutual information based feature selection , 2014, KDD.

[23]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[24]  Jingcheng Wang,et al.  Neighborhood effective information ratio for hybrid feature subset evaluation and selection , 2013, Neurocomputing.

[25]  Xiao Zhang,et al.  Feature selection in mixed data: A method using a novel fuzzy rough set-based information entropy , 2016, Pattern Recognit..

[26]  Ahmed Al-Ani,et al.  A dependency-based search strategy for feature selection , 2009, Expert Syst. Appl..

[27]  James Bailey,et al.  Can high-order dependencies improve mutual information based feature selection? , 2016, Pattern Recognit..

[28]  Gang Chen,et al.  A novel wrapper method for feature selection and its applications , 2015, Neurocomputing.

[29]  Mark A. Hall,et al.  Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning , 1999, ICML.

[30]  Wenhao Shu,et al.  Mutual information criterion for feature selection from incomplete data , 2015, Neurocomputing.

[31]  Rossitza Setchi,et al.  Feature selection using Joint Mutual Information Maximisation , 2015, Expert Syst. Appl..

[32]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[33]  Huan Liu,et al.  Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution , 2003, ICML.

[34]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[35]  Jiye Liang,et al.  An efficient feature selection algorithm for hybrid data , 2016, Neurocomputing.

[36]  Zexuan Zhu,et al.  Wrapper–Filter Feature Selection Algorithm Using a Memetic Framework , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[37]  Jose Miguel Puerta,et al.  Speeding up incremental wrapper feature subset selection with Naive Bayes classifier , 2014, Knowl. Based Syst..

[38]  Min Du,et al.  cDNA microarray adaptive segmentation , 2014, Neurocomputing.

[39]  Xindong Wu,et al.  Quality of information-based source assessment and selection , 2014, Neurocomputing.

[40]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[41]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Randy J. Pell,et al.  Genetic algorithms combined with discriminant analysis for key variable identification , 2004 .

[43]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[44]  Qinghua Hu,et al.  Neighborhood rough set based heterogeneous feature subset selection , 2008, Inf. Sci..

[45]  Xindong Wu,et al.  Feature selection using hierarchical feature clustering , 2011, CIKM '11.

[46]  Jinkun Chen,et al.  Feature selection via neighborhood multi-granulation fusion , 2014, Knowl. Based Syst..

[47]  Naftali Tishby,et al.  Margin based feature selection - theory and algorithms , 2004, ICML.

[48]  Haleh Vafaie,et al.  Feature Selection Methods: Genetic Algorithms vs. Greedy-like Search , 2009 .

[49]  Thomas G. Dietterich,et al.  Learning with Many Irrelevant Features , 1991, AAAI.

[50]  Zehang Sun,et al.  Object detection using feature subset selection , 2004, Pattern Recognit..

[52]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[53]  Aixia Guo,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2014 .