A multi-dimensional measure function for classifier performance

Evaluation of classifier performance is often based on statistical methods e.g. cross-validation tests. In these tests performance is often strongly related to or solely based on the accuracy of the classifier on a limited set of instances. The use of measure functions has been suggested as a promising approach to deal with this limitation. However, no usable implementation of a measure function has yet been presented. This article presents such an implementation and demonstrates its usage through a set of experiments. The results indicate that there are cases for which measure functions may be able to capture important aspects of the evaluated classifier that cannot be captured by cross-validation tests.

[1]  Robert Swan. Sturtevant,et al.  Bulletin of the American Iris Society , 1931 .

[2]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[3]  Michael Georgiopoulos,et al.  Improving generalization by using genetic algorithms to determine the neural network size , 1995, Proceedings of Southcon '95.

[4]  Tom M. Mitchell,et al.  Machine learning, International Edition , 1997, McGraw-Hill Series in Computer Science.

[5]  Paul Davidsson,et al.  Measuring generalization quality , 1998 .

[6]  M. Mcdermott,et al.  Selecting a Generic Measure of Health-Related Quality of Life for Use among Older Adults , 1998, Evaluation & the health professions.

[7]  Stephan K. Chalup,et al.  A study on hill climbing algorithms for neural network training , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[8]  Paul Davidsson,et al.  Measure-based classifier performance evaluation , 1999, Pattern Recognit. Lett..

[9]  Niall M. Adams,et al.  Improving the Practice of Classifier Performance Assessment , 2000, Neural Computation.

[10]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[11]  Marcus A. Maloof.  On machine learning, ROC analysis, and statistical tests of significance , 2002, Object recognition supported by user interaction for service robots.

[12]  Shinn-Ying Ho,et al.  Design of an optimal nearest neighbor classifier using an intelligent genetic algorithm , 2002, Pattern Recognit. Lett..

[13]  Niklas Lavesson.  Evaluation of classifier performance and the impact of learning algorithm parameters , 2003 .