A New Disagreement Measure for Characterization of Classification Problems

Robert P.W. Duin, Elzbieta Pekalska and David M.J. Tax proposed the characterization of classification problems by classifier disagreement. They showed that it is possible to use a standard set of supervised classification problems for constructing a rule that allows deciding about the similarity of new problems to the existing ones. The classifier disagreement could be used to group classification problems in a way which could help to select the appropriate tools for solving new problems. Duin et al proposed a dissimilarity measure between two problems taking into account only the full disagreement matrices. They used a measure of the disagreement based on the coincidence of the classifier output however the correctness was not considered. In this work, we propose a new measure of disagreement which takes into account the correctness of classification result. To calculate the disagreement each object is analyzed to verify if it was classified correctly or incorrectly by the classifiers. We use this new disagreement measure to calculate the dissimilarity between two problems. Some experiments were done and the results were compared against Duin’s et al results.