Confusion and Distance Metrics as Performance Criteria for Hierarchical Classification Spaces

When intelligent systems reason about complex problems with a large hierarchical classification space it is hard to evaluate system performance. For classification problems, different evaluation criteria exist but these either focus on a belief expressed on all possible, mutually exclusive labels (soft classification) or they are based on the set of labels that are returned by a classifier (hard classification) for hierarchical labels. Measures to valuate a classifier that assigns belief on all labels when these are hierarchical related however are lacking. This paper puts forward two new criteria for evaluation of soft output for hierarchical labels using a generic and flexible model of the solution space. The first criterion gives information on the accuracy of the system and the second on the robustness. Results with these new criteria are compared to existing criteria for a hierarchical classification task with different classifiers.

[1]  Thomas D. Nielsen,et al.  Classification using Hierarchical Naïve Bayes models , 2006, Machine Learning.

[2]  D. Joanes,et al.  Comparing measures of sample skewness and kurtosis , 1998 .

[3]  Claudio Gentile,et al.  Incremental Algorithms for Hierarchical Classification , 2004, J. Mach. Learn. Res..

[4]  T. Subba Rao,et al.  Classification, Parameter Estimation and State Estimation: An Engineering Approach Using MATLAB , 2004 .

[5]  Thomas Hofmann,et al.  Hierarchical document categorization with support vector machines , 2004, CIKM '04.

[6]  Wilbert van Norden,et al.  Applying the PCR6 Rule of combination in real time classification systems , 2009, 2009 12th International Conference on Information Fusion.

[7]  Yiming Yang,et al.  An Evaluation of Statistical Approaches to Text Categorization , 1999, Information Retrieval.

[8]  Jean Dezert,et al.  An Introduction to the DSm Theory for the Combination of Paradoxical, Uncertain, and Imprecise Sources of Information , 2006, ArXiv.

[9]  Christophe Osswald,et al.  A new generalization of the proportional conflict redistribution rule stable in terms of decision , 2008, ArXiv.

[10]  Yoram Singer,et al.  Large margin hierarchical classification , 2004, ICML.

[11]  Catholijn M. Jonker,et al.  Classification support using confidence intervals , 2008, 2008 11th International Conference on Information Fusion.

[12]  Florentin Smarandache,et al.  Advances and Applications of DSmT for Information Fusion (Collected Works) , 2004 .

[13]  Juho Rousu,et al.  Kernel-Based Learning of Hierarchical Multilabel Classification Models , 2006, J. Mach. Learn. Res..

[14]  Michael J. Witbrock,et al.  Autonomous Classification of Knowledge into an Ontology , 2007, FLAIRS.