ROC analysis of classifiers in machine learning: A survey

The use of ROC Receiver Operating Characteristics analysis as a tool for evaluating the performance of classification models in machine learning has been increasing in the last decade. Among the most notable advances in this area are the extension of two-class ROC analysis to the multi-class case as well as the employment of ROC analysis in cost-sensitive learning. Methods now exist which take instance-varying costs into account. The purpose of our paper is to present a survey of this field with the aim of gathering important achievements in one place. In the paper, we present application areas of the ROC analysis in machine learning, describe its problems and challenges and provide a summarized list of alternative approaches to ROC analysis. In addition to presented theory, we also provide a couple of examples intended to illustrate the described approaches.

[1]  Kent A. Spackman,et al.  Signal Detection Theory: Valuable Tools for Evaluating Inductive Learning , 1989, ML.

[2]  José Hernández-Orallo,et al.  Volume under the ROC Surface for Multi-class Problems , 2003, ECML.

[3]  Eyke Hüllermeier,et al.  A critical analysis of variants of the AUC , 2008, Machine Learning.

[4]  Tom Fawcett,et al.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions , 1997, KDD.

[5]  Remco R. Bouckaert,et al.  Efficient AUC Learning Curve Calculation , 2006, Australian Conference on Artificial Intelligence.

[6]  Peter A. Flach,et al.  Improving Accuracy and Cost of Two-class and Multi-class Probabilistic Classifiers Using ROC Curves , 2003, ICML.

[7]  Geoffrey I. Webb,et al.  On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions , 2005, Machine Learning.

[8]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[9]  øöö Blockinøø.  Well-Trained PETs : Improving Probability Estimation , 2000 .

[10]  Robert C. Holte,et al.  What ROC Curves Can't Do (and Cost Curves Can) , 2004, ROCAI.

[11]  Robert P. W. Duin,et al.  Learning Curves for the Analysis of Multiple Instance Classifiers , 2008, SSPR/SPR.

[12]  Johannes Fürnkranz,et al.  An Analysis of Rule Evaluation Metrics , 2003, ICML.

[13]  Foster J. Provost,et al.  Confidence Bands for Roc Curves , 2004, ROCAI.

[14]  Alvin F. Martin,et al.  The DET curve in assessment of detection task performance , 1997, EUROSPEECH.

[15]  Tom Fawcett,et al.  ROC Graphs: Notes and Practical Considerations for Data Mining Researchers , 2003 .

[16]  Peter A. Flach,et al.  Decision Support for Data Mining , 2003 .

[17]  Niall M. Adams,et al.  An improved measure for comparing diagnostic tests , 2000, Comput. Biol. Medicine.

[18]  Kevin W. Bowyer,et al.  Generating ROC curves for artificial neural networks , 1994, IEEE Transactions on Medical Imaging.

[19]  Hendrik Blockeel,et al.  Deriving biased classifiers for better ROC performance , 2002, Informatica.

[20]  Miha Vuk,et al.  ROC curve, lift chart and calibration plot , 2006, Advances in Methodology and Statistics.

[21]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[22]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[23]  D. Mossman.  Three-way ROCs , 1999, Medical decision making : an international journal of the Society for Medical Decision Making.

[24]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[25]  Kun Deng,et al.  Active Learning to Maximize Area Under the ROC Curve , 2006, Sixth International Conference on Data Mining (ICDM'06).

[26]  Ashwin Srinivasan,et al.  Extracting Context-Sensitive Models in Inductive Logic Programming , 2001, Machine Learning.

[27]  Ross Bettinger,et al.  Cost-Sensitive Classifier Selection Using the ROC Convex Hull Method , 2022 .

[28]  Jonathan E. Fieldsend,et al.  Multi-class ROC analysis from a multi-objective optimisation perspective , 2006, Pattern Recognit. Lett..

[29]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[30]  Lei Ding,et al.  Error analysis of classifiers in machine learning , 2010, 2010 3rd International Congress on Image and Signal Processing.

[31]  James P. Egan,et al.  Signal detection theory and ROC analysis , 1975 .

[32]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[33]  Charles X. Ling,et al.  Partial ensemble classifiers selection for better ranking , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[34]  Robert C. Holte,et al.  Cost curves: An improved method for visualizing classifier performance , 2006, Machine Learning.

[35]  Francesco Tortorella,et al.  A ROC-Based Reject Rule for Support Vector Machines , 2003, MLDM.

[36]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[37]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[38]  Szymon Jaroszewicz,et al.  Efficient AUC Optimization for Classification , 2007, PKDD.

[39]  Stephan Dreiseitl,et al.  Training Multiclass Classifiers by Maximizing the Volume Under the ROC Surface , 2007, EUROCAST.

[40]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[41]  Peter A. Flach,et al.  Data Mining and Decision Support: Aspects of Integration and Collaboration , 2003 .

[42]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[43]  T. Koepsell,et al.  Epidemiologic Methods: Studying the Occurrence of Illness , 2003 .

[44]  Peter A. Flach,et al.  Modifying ROC Curves to Incorporate Predicted Probabilities , 2005 .

[45]  Kevin S. Woods.  Computer-Aided Diagnosis and Automated Screening of Digital Mammogram. , 1997 .

[46]  Peter A. Flach,et al.  Learning Decision Trees Using the Area Under the ROC Curve , 2002, ICML.

[47]  D. Bamber.  The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[48]  Robert C. Holte,et al.  Explicitly representing expected cost: an alternative to ROC representation , 2000, KDD '00.

[49]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[50]  Tom Fawcett,et al.  ROC graphs with instance-varying costs , 2006, Pattern Recognit. Lett..

[51]  José Hernández-Orallo,et al.  An experimental comparison of performance measures for classification , 2009, Pattern Recognit. Lett..

[52]  N. Obuchowski.  Receiver operating characteristic curves and their use in radiology. , 2003, Radiology.

[53]  Peter A. Flach,et al.  A Response to Webb and Ting’s On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions , 2005, Machine Learning.

[54]  Johannes Fürnkranz,et al.  An Analysis of Rule Learning Heuristics , 2003 .

[55]  Peter A. Flach,et al.  An Improved Model Selection Heuristic for AUC , 2007, ECML.

[56]  Peter A. Flach.  The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics , 2003, ICML.

[57]  Moisés Goldszmidt,et al.  Properties and Benefits of Calibrated Classifiers , 2004, PKDD.

[58]  Terran Lane,et al.  Extensions of ROC Analysis to multi-class domains , 2000 .

[59]  Niall M. Adams,et al.  Comparing classifiers when the misallocation costs are uncertain , 1999, Pattern Recognit..

[60]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[61]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[62]  Robert P. W. Duin,et al.  Approximating the multiclass ROC by pairwise analysis , 2007, Pattern Recognit. Lett..