An instance-oriented performance measure for classification

Abstract Performance evaluation is significant in data classification. The existing evaluation methods ignore the characteristics (such as classification difficulty) of each instance. In practice, it is necessary to measure classification performance from the perspective of instances. In this paper, an instance-oriented classification performance metric is proposed based on the classification difficulty of each instance, named degree of credibility (Cr ). Cr conforms to the natural cognition that the lower the probability of misclassifying relatively easy instances, the more credible the classifier. It focuses on the credibility of each instance’s prediction, which opens up a new way for classifier evaluation. Moreover, several important properties of Cr are identified, laying solid theoretical foundation for classifier evaluation. Also, the concept of acceptable classifier is proposed to judge whether the trained model and its parameter set reach excellent ranks at the current technology level instead of relying entirely on human experience. The experimental results of twelve classifiers on twelve datasets indicate the physical significance and good statistical consistency and discriminatory ability of Cr, as well as the feasibility of acceptable classifiers for model selection and training. Furthermore, the proposal of approximate difficulty greatly improves the computation efficiency of instance difficulty.