HMC-ReliefF: Feature ranking for hierarchical multi-label classification

In machine learning, the growing complexity of the available data poses an increased challenge for its analysis. The rising complexity is both in terms of the data becoming more high-dimensional as well as the data having a more intricate structure. This emphasizes the need for developing machine learning algorithms that are able to tackle both the high-dimensionality and the complex structure of the data. Our work in this paper focuses on the development and analysis of the HMCReliefF algorithm, which is a feature relevance (ranking) algorithm for the task of Hierarchical Multi-label Classification (HMC). The basis of the algorithm is the RReliefF algorithm for regression that is adapted for hierarchical multi-label target variables. We perform an extensive experimental investigation of the HMC-ReliefF algorithm on several datasets from the domains of image annotation and functional genomics. We analyse the algorithm’s performance in terms of accuracy in a filterlike setting and also in terms of ranking stability for various parameter values. The results show that the HMC-ReliefF can successfully detect relevant features from the data that can be further used for constructing accurate predictive models. Additionally, the stability analysis helps to determine the preferred parameter values for obtaining not just accurate, but also a stable algorithm output.

[1]  Thomas G. Dietterich,et al.  Structured machine learning: the next ten years , 2008, Machine Learning.

[2]  Peerapon Vateekul,et al.  Hierarchical Multi-Label Classification: Going Beyond Generalization Trees , 2012 .

[3]  Michel Verleysen,et al.  Feature Selection for Multi-label Classification Problems , 2011, IWANN.

[4]  Hans-Peter Kriegel,et al.  Future trends in data mining , 2007, Data Mining and Knowledge Discovery.

[5]  Marko Robnik-Sikonja,et al.  An adaptation of Relief for attribute estimation in regression , 1997, ICML.

[6]  Marko Robnik-Sikonja,et al.  Theoretical and Empirical Analysis of ReliefF and RReliefF , 2003, Machine Learning.

[7]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[8]  Saso Dzeroski,et al.  Hierarchical annotation of medical images , 2011, Pattern Recognit..

[9]  Hugh E. Williams,et al.  Simple and accurate feature selection for hierarchical categorisation , 2002, DocEng '02.

[10]  Cesare Furlanello,et al.  Algebraic stability indicators for ranked lists in molecular profiling , 2008, Bioinform..

[11]  Newton Spolaôr,et al.  ReliefF for Multi-label Feature Selection , 2013, 2013 Brazilian Conference on Intelligent Systems.

[12]  Sašo Džeroski,et al.  Extending ReliefF for Hierarchical Multi-label Classification ? , 2013 .

[13]  G. N. Lance,et al.  Mixed-Data Classificatory Programs I - Agglomerative Systems , 1967, Aust. Comput. J..

[14]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[15]  Zengyou He,et al.  Stable Feature Selection for Biomarker Discovery , 2010, Comput. Biol. Chem..

[16]  Saso Dzeroski,et al.  Decision trees for hierarchical multi-label classification , 2008, Machine Learning.

[17]  Geoff Holmes,et al.  Multi-label Classification Using Ensembles of Pruned Sets , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[18]  Peter D. Turney.  Technical note: Bias and the quantification of stability , 1995, Machine Learning.

[19]  Qiang Wang,et al.  A Novel Feature Selection Method Based on Category Information Analysis for Class Prejudging in Text Classification , 2006 .

[20]  Saso Dzeroski,et al.  Predicting gene function using hierarchical multi-label decision tree ensembles , 2010, BMC Bioinformatics.

[21]  Thomas Martin Deserno,et al.  Automatic medical image annotation in ImageCLEF 2007: Overview, results, and discussion , 2008, Pattern Recognit. Lett..

[22]  Newton Spolaôr,et al.  Feature Selection for Multi-label Learning: A Systematic Literature Review and Some Experimental Evaluations , 2015, Int. J. Comput. Intell. Syst..

[23]  Zhong Ming,et al.  Text Learning and Hierarchical Feature Selection in Webpage Classification , 2008, ADMA.

[24]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[25]  Grigorios Tsoumakas,et al.  Mining Multi-label Data , 2010, Data Mining and Knowledge Discovery Handbook.

[26]  Alex A. Freitas,et al.  A survey of hierarchical classification across different application domains , 2010, Data Mining and Knowledge Discovery.

[27]  Dragi Kocev,et al.  Evaluation of Distance Measures for Hierarchical Multi-Label Classification in Functional Genomics , 2015 .

[28]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[29]  Newton Spolaôr,et al.  A Comparison of Multi-label Feature Selection Methods using the Problem Transformation Approach , 2013, CLEI Selected Papers.

[30]  Saso Dzeroski,et al.  Tree ensembles for predicting structured outputs , 2013, Pattern Recognit..

[31]  Bingru Yang,et al.  Hierarchical Text Categorization Based on Multiple Feature Selection and Fusion of Multiple Classifiers Approaches , 2009, 2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery.

[32]  P. Jaccard,et al.  Etude comparative de la distribution florale dans une portion des Alpes et des Jura , 1901 .

[33]  Catia Pesquita,et al.  Evaluating GO-based Semantic Similarity Measures , 2007 .

[34]  Saso Dzeroski,et al.  Hierarchical classification of diatom images using ensembles of predictive clustering trees , 2012, Ecol. Informatics.

[35]  Saso Dzeroski,et al.  ReliefF for Hierarchical Multi-label Classification , 2013, NFMCP.

[36]  Andrea Esuli,et al.  TreeBoost.MH: A Boosting Algorithm for Multi-label Hierarchical Text Categorization , 2006, SPIRE.

[37]  Thomas Gärtner,et al.  On structured output training: hard cases and an efficient alternative , 2009, Machine Learning.

[38]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[39]  M. Ashburner,et al.  Gene Ontology: tool for the unification of biology , 2000, Nature Genetics.

[40]  G. N. Lance,et al.  Computer Programs for Hierarchical Polythetic Classification ("Similarity Analyses") , 1966, Comput. J..