Entropy Criterion for Classifier-Independent Feature Selection

Feature selection aims to select a feature subset that has discriminative information from the original feature set. In practice, we do not know what classifier is used beforehand, and it is preferable to find a feature subset that is universally effective for any classifier. Such a trial is called classifier-independent feature selection and can be made by removing garbage features that have no discriminative information. However, it is difficult to distinguish only garbage features from the others. In this study, we propose an entropy criterion for this goal and confirm the effectiveness through a synthetic dataset.

[1]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[2]  Mineichi Kudo,et al.  Construction of class regions by a randomized algorithm: a randomized subclass method , 1996, Pattern Recognit..

[3]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[4]  Francesc J. Ferri,et al.  Comparative study of techniques for large-scale feature selection* *This work was suported by a SERC grant GR/E 97549. The first author was also supported by a FPI grant from the Spanish MEC, PF92 73546684 , 1994 .

[5]  Jerome H. Friedman,et al.  A Recursive Partitioning Decision Rule for Nonparametric Classification , 1977, IEEE Transactions on Computers.

[6]  Samy Bengio,et al.  SVMTorch: Support Vector Machines for Large-Scale Regression Problems , 2001, J. Mach. Learn. Res..

[7]  Sameer Singh,et al.  PRISM – A novel framework for pattern recognition , 2003, Pattern Analysis & Applications.

[8]  Mineichi Kudo,et al.  Feature selection based on the structural indices of categories , 1993, Pattern Recognit..

[9]  Josef Kittler,et al.  Divergence Based Feature Selection for Multimodal Class Densities , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[11]  M. Loew,et al.  Relative feature importance: A classifier-independent approach to feature selection , 1994 .

[12]  Mineichi Kudo,et al.  Comparison of algorithms that select features for pattern classifiers , 2000, Pattern Recognit..