Performance measures for classification systems with rejection

Classifiers with rejection are essential in real-world applications where misclassifications and their effects are critical. However, if no problem specific cost function is defined, there are no established measures to assess the performance of such classifiers. We introduce a set of desired properties for performance measures for classifiers with rejection, based on which we propose a set of three performance measures for the evaluation of the performance of classifiers with rejection that satisfy the desired properties. The nonrejected accuracy measures the ability of the classifier to accurately classify nonrejected samples; the classification quality measures the correct decision making of the classifier with rejector; and the rejection quality measures the ability to concentrate all misclassified samples onto the set of rejected samples. From the measures, we derive the concept of relative optimality that allows us to connect the measures to a family of cost functions that take into account the trade-off between rejection and misclassification. We illustrate the use of the proposed performance measures on classifiers with rejection applied to synthetic and real-world data.

[1]  C. K. Chow,et al.  On optimum recognition error and reject tradeoff , 1970, IEEE Trans. Inf. Theory.

[2]  Yves Grandvalet,et al.  Support Vector Machines with a Reject Option , 2008, NIPS.

[3]  Blaise Hanczar,et al.  Classification with reject option in gene expression data , 2008, Bioinform..

[4]  Jelena Kovacevic,et al.  Supervised hyperspectral image classification with rejection , 2016, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[5]  Rafael Llobet,et al.  Batch-adaptive rejection threshold estimation with application to OCR post-processing , 2015, Expert Syst. Appl..

[6]  Fabio Roli,et al.  A Classification Approach with a Reject Option for Multi-label Problems , 2011, ICIAP.

[7]  L. Mascarilla,et al.  A class of reject-first possibilistic classifiers , 2001, Proceedings Joint 9th IFSA World Congress and 20th NAFIPS International Conference (Cat. No. 01TH8569).

[8]  Camelia Chira,et al.  Classifiers with a reject option for early time-series classification , 2013, 2013 IEEE Symposium on Computational Intelligence and Ensemble Learning (CIEL).

[9]  Jelena Kovacevic,et al.  Alternating direction optimization for image segmentation using hidden Markov measure field models , 2014, Electronic Imaging.

[10]  Michael Biehl,et al.  Matrix relevance LVQ in steroid metabolomics based classification of adrenal tumors , 2012, ESANN.

[11]  Fabio Roli,et al.  Support Vector Machines with Embedded Reject Option , 2002, SVM.

[12]  Jelena Kovacevic,et al.  Supervised hyperspectral image segmentation: A convex formulation using hidden fields , 2014, 2014 6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[13]  Fabio Roli,et al.  Classification with reject option in text categorisation systems , 2003, 12th International Conference on Image Analysis and Processing, 2003.Proceedings..

[14]  S CardosoJaime,et al.  The data replication method for the classification with reject option , 2013 .

[15]  Fabio Roli,et al.  Reject option with multiple thresholds , 2000, Pattern Recognit..

[16]  Ming Yuan,et al.  Classification Methods with Reject Option Based on Convex Risk Minimization , 2010, J. Mach. Learn. Res..

[17]  Jelena Kovacevic,et al.  Classification with reject option using contextual information , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[18]  Hoel Le Capitaine.  A unified view of class-selection with probabilistic classifiers , 2014, Pattern Recognit..

[19]  Reinhold Huber-Mörk,et al.  Classification of coins using an eigenspace approach , 2005, Pattern Recognit. Lett..

[20]  Francesco Tortorella.  An Optimal Reject Rule for Binary Classifiers , 2000, SSPR/SPR.

[21]  Laurent Mascarilla,et al.  A k-order fuzzy OR operator for pattern classification with k-order ambiguity rejection , 2008, Fuzzy Sets Syst..

[22]  Tadeusz Pietraszek,et al.  On the use of ROC analysis for the optimization of abstaining classifiers , 2007, Machine Learning.

[23]  Sameer Singh,et al.  Indoor vs. outdoor scene classification in digital photographs , 2005, Pattern Recognit..

[24]  Frans Coenen,et al.  Highly reliable breast cancer diagnosis with cascaded ensemble classifiers , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[25]  Fabio Roli,et al.  Multi-label classification with a reject option , 2013, Pattern Recognit..

[26]  Ajalmar R. da Rocha Neto,et al.  Classification with reject option for software defect prediction , 2016, Appl. Soft Comput..

[27]  Jaime S. Cardoso,et al.  Diagnostic of Pathology on the Vertebral Column with Embedded Reject Option , 2011, IbPRIA.

[28]  Robert P. W. Duin,et al.  The interaction between classification and reject performance for distance-based reject-option classifiers , 2006, Pattern Recognit. Lett..

[29]  L. Mascarilla,et al.  A third way to design pattern classifiers with reject options , 2002, 2002 Annual Meeting of the North American Fuzzy Information Processing Society Proceedings. NAFIPS-FLINT 2002 (Cat. No. 02TH8622).

[30]  Hoel Le Capitaine,et al.  A family of measures for best top-n class-selective decision rules , 2012, Pattern Recognit..

[31]  Jaime S. Cardoso,et al.  The data replication method for the classification with reject option , 2010, AI Commun..

[32]  Jelena Kovacevic,et al.  Robust hyperspectral image classification with rejection fields , 2015, 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[33]  R. Boqué,et al.  Classification from microarray data using probabilistic discriminant partial least squares with reject option. , 2009, Talanta.

[34]  Heiko Wersing,et al.  Efficient rejection strategies for prototype-based classification , 2015, Neurocomputing.

[35]  Francesco Tortorella,et al.  Reducing the classification cost of support vector classifiers through an ROC-based reject rule , 2004, Pattern Analysis and Applications.

[36]  Yee-Hong Yang,et al.  Classifier design with incomplete knowledge , 1998, Pattern Recognit..

[37]  Michael Biehl,et al.  Urine Steroid Metabolomics as a Biomarker Tool for Detecting Malignancy in Adrenal Tumors , 2011, The Journal of clinical endocrinology and metabolism.

[38]  Robert P. W. Duin,et al.  Growing a multi-class classifier with a reject option , 2008, Pattern Recognit. Lett..

[39]  Carla M. Santos-Pereira,et al.  On optimal reject rules and ROC curves , 2005, Pattern Recognit. Lett..

[40]  Anil K. Jain,et al.  Image classification for content-based indexing , 2001, IEEE Trans. Image Process..

[41]  Blaise Hanczar,et al.  Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option , 2009, MLSB.

[42]  J. Chanussot,et al.  Hyperspectral Remote Sensing Data Analysis and Future Challenges , 2013, IEEE Geoscience and Remote Sensing Magazine.

[43]  Harold Mouchère,et al.  A Unified Strategy to Deal with Different Natures of Reject , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[44]  José Ramón Quevedo,et al.  Disease Liability Prediction from Large Scale Genotyping Data Using Classifiers with a Reject Option , 2012, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[45]  Bernard Dubuisson,et al.  A statistical decision rule with incomplete knowledge about classes , 1993, Pattern Recognit..

[46]  Lorenzo Bruzzone,et al.  Combination of neural and statistical algorithms for supervised classification of remote-sensing image , 2000, Pattern Recognit. Lett..

[47]  Radu Herbei,et al.  Classification with reject option , 2006 .

[48]  Francesco Tortorella,et al.  A ROC-based reject rule for dichotomizers , 2005, Pattern Recognit. Lett..

[49]  Fabio Roli,et al.  Analysis of error-reject trade-off in linearly combined multiple classifiers , 2004, Pattern Recognit..

[50]  Laurent Mascarilla,et al.  Reject Strategies Driven Combination of Pattern Classifiers , 2002, Pattern Analysis & Applications.

[51]  Claudio Marrocco,et al.  Design of reject rules for ECOC classification systems , 2012, Pattern Recognit..

[52]  M. Wegkamp.  Lasso type classifiers with a reject option , 2007, 0705.2363.

[53]  Michael Biehl,et al.  Adaptive Relevance Matrices in Learning Vector Quantization , 2009, Neural Computation.

[54]  Blaise Hanczar,et al.  Combination of One-Class Support Vector Machines for Classification with Reject Option , 2014, ECML/PKDD.

[55]  Mario Vento,et al.  Multiclassification: reject criteria for the Bayesian combiner , 1999, Pattern Recognit..

[56]  Heiko Wersing,et al.  Optimal local rejection for classifiers , 2016, Neurocomputing.

[57]  Antonio J. Plaza,et al.  Hyperspectral Image Segmentation Using a New Bayesian Approach With Active Learning , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Fabio Roli,et al.  A Two-Stage Classifier with Reject Option for Text Categorisation , 2004, SSPR/SPR.

[59]  Hoel Le Capitaine,et al.  A new fuzzy 3-rules pattern classifier with reject options based on aggregation of membership degrees , 2008 .

[60]  Atsushi Sato,et al.  Generalized Learning Vector Quantization , 1995, NIPS.