The Multiclass ROC Front method for cost-sensitive classification

This paper addresses the problem of learning a multiclass classification system that can suit to any environment. By that we mean that particular (imbalanced) misclassification costs are taken into account by the classifier for predictions. However, these costs are not well known during the learning phase in most cases, or may evolve afterwards. There is a need in that case to learn a classifier that can potentially suit to any of these costs in prediction phase. The learning method proposed in this work, named the Multiclass ROC Front (MROCF) method, responds to this issue by exploiting ROC-based tools through a multiobjective optimization process. While this type of ROC-based multiobjective optimization approach has been successfully used for two-class problems, it has never been proposed in real-world multiclass classification problems. Experiments led on several real-world datasets show that the MROCF method offers a major improvement over a cost-insensitive classifier and is competitive with the state-of-the-art cost-sensitive optimization method on all but one of the 20 datasets. HighlightsWe propose a new method for multiclass cost-sensitive classification when misclassification costs are unknown during training.It is based on a multi-model approach and can suit to any cost-sensitive environment in prediction.It makes use of ROC-based multi-objective optimization algorithms.The method is compared to a cost-insensitive method and a state-of-the-art cost-sensitive optimization method.It outperforms both methods for most of the datasets tested.

[1]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[2]  Christian Gagné,et al.  Multi-objective evolutionary optimization for generating ensembles of classifiers in the ROC space , 2012, GECCO '12.

[3]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[4]  Sunil Vadera,et al.  A survey of cost-sensitive decision tree induction algorithms , 2013, CSUR.

[5]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[6]  Robert Sabourin,et al.  Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs , 2010, Pattern Recognit..

[7]  Robert C. Holte,et al.  C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats Over-Sampling , 2003 .

[8]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[9]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[10]  Ulf Brefeld,et al.  {AUC} maximizing support vector learning , 2005 .

[11]  Yves Lecourtier,et al.  A multi-model selection framework for unknown and/or evolutive misclassification cost problems , 2010, Pattern Recognit..

[12]  Peter A. Flach,et al.  Learning Decision Trees Using the Area Under the ROC Curve , 2002, ICML.

[13]  Jonathan E. Fieldsend,et al.  Multiobjective Supervised Learning , 2008, Multiobjective Problem Solving from Nature.

[14]  Robert E. Schapire,et al.  On reoptimizing multi-class classifiers , 2008, Machine Learning.

[15]  Pavel Paclík,et al.  The ROC skeleton for multiclass ROC estimation , 2010, Pattern Recognit. Lett..

[16]  Dazhe Zhao,et al.  An Optimized Cost-Sensitive SVM for Imbalanced Data Learning , 2013, PAKDD.

[17]  Peter A. Flach,et al.  Improving Accuracy and Cost of Two-class and Multi-class Probabilistic Classifiers Using ROC Curves , 2003, ICML.

[18]  Chih-Jen Lin,et al.  Probability Estimates for Multi-class Classification by Pairwise Coupling , 2003, J. Mach. Learn. Res..

[19]  Matthew A. Kupinski,et al.  Multiobjective Genetic Optimization of Diagnostic Classifiers with Implications for Generating ROC Curves , 1999, IEEE Trans. Medical Imaging.

[20]  Aravind Seshadri,et al.  A FAST ELITIST MULTIOBJECTIVE GENETIC ALGORITHM: NSGA-II , 2000 .

[21]  Alain Rakotomamonjy,et al.  Optimizing Area Under Roc Curve with SVMs , 2004, ROCAI.

[22]  Jonathan E. Fieldsend,et al.  Multi-class ROC analysis from a multi-objective optimisation perspective , 2006, Pattern Recognit. Lett..

[23]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[24]  José Hernández-Orallo,et al.  Volume under the ROC Surface for Multi-class Problems , 2003, ECML.

[25]  Tom Fawcett,et al.  ROC Graphs: Notes and Practical Considerations for Researchers , 2007 .

[26]  Dmitry O. Gorodnichy,et al.  Skew-sensitive boolean combination for adaptive ensembles - An application to face recognition in video surveillance , 2014, Inf. Fusion.

[27]  Laurent Heutte,et al.  Using Random Forests for Handwritten Digit Recognition , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[28]  Hang Li,et al.  Cost-Sensitive Learning of SVM for Ranking , 2006, ECML.

[29]  Robert P. W. Duin,et al.  A simplified extension of the Area under the ROC to the multiclass domain , 2006 .

[30]  Robert M. Nishikawa,et al.  Optimization and FROC analysis of rule-based detection schemes using a multiobjective approach , 1998, IEEE Transactions on Medical Imaging.

[31]  Ross A. McDonald,et al.  The mean subjective utility score, a novel metric for cost-sensitive classifier evaluation , 2006, Pattern Recognition Letters.

[32]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[33]  David E. Goldberg,et al.  A niched Pareto genetic algorithm for multiobjective optimization , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[34]  Robert P. W. Duin,et al.  Approximating the multiclass ROC by pairwise analysis , 2007, Pattern Recognit. Lett..