Feature selection considering two types of feature relevancy and feature interdependency

A novel feature selection method is proposed based on information theory.Our method divides feature relevancy into two categories.We performed experiments over 12 public data sets.Our method outperforms five competing methods in terms of accuracy.Our method selects few number of features when it achieves the highest accuracy. Feature selection based on information theory, which is used to select a group of the most informative features, has extensive application fields such as machine learning, data mining and natural language processing. However, numerous previous methods suffer from two common defects. (1) Feature relevancy is defined without distinguishing candidate feature relevancy and selected feature relevancy. (2) Some interdependent features may be misinterpreted as redundant features. In this study, we propose a feature selection method named Dynamic Relevance and Joint Mutual Information Maximization (DRJMIM) to address these two defects. DRJMIM includes four stages. First, the relevancy is divided into two categories: candidate feature relevancy and selected feature relevancy. Second, according to candidate feature relevancy that is joint mutual information, some redundant features are selected. Third, the redundant features are combined with a dynamic weight to reduce the selection possibility of true redundant features while increasing the false ones. Finally, the most informative and interdependent features are selected and true redundant features are eliminated simultaneously. Furthermore, our method is compared with five competitive feature selection methods on 12 publicly available data sets. The classification results show that DRJMIM performs better than other five methods. Its statistical significance is verified by a paired two-tailed t-test. Meanwhile, DRJMIM obtains few number of selected features when it achieves the highest classification accuracy.