ROC Graphs: Notes and Practical Considerations for Researchers

Receiver Operating Characteristics (ROC) graphs are a useful technique for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been increasingly adopted in the machine learning and data mining research communities. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. This article serves both as a tutorial introduction to ROC graphs and as a practical guide for using them in research.

[1]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[2]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[3]  V. Rich.  Personal communication , 1989, Nature.

[4]  Kent A. Spackman,et al.  Signal Detection Theory: Valuable Tools for Evaluating Inductive Learning , 1989, ML.

[5]  David D. Lewis,et al.  Representation Quality in Text Classification: An Introduction and Experiment , 1990, HLT.

[6]  David D. Lewis,et al.  Evaluating Text Categorization I , 1991, HLT.

[7]  S. Clearwater,et al.  A rule-learning program in high energy physics event classification , 1991 .

[8]  Tom Fawcett,et al.  Combining Data Mining and Machine Learning for Effective User Profiling , 1996, KDD.

[9]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[10]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[11]  Alvin F. Martin,et al.  The DET curve in assessment of detection task performance , 1997, EUROSPEECH.

[12]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[13]  Tom Fawcett,et al.  Robust Classification Systems for Imprecise Environments , 1998, AAAI/IAAI.

[14]  Pedro M. Domingos.  MetaCost: a general method for making classifiers cost-sensitive , 1999, KDD '99.

[15]  Terran Lane,et al.  Extensions of ROC Analysis to multi-class domains , 2000 .

[16]  øöö Blockinøø.  Well-Trained PETs : Improving Probability Estimation , 2000 .

[17]  Robert C. Holte,et al.  Explicitly representing expected cost: an alternative to ROC representation , 2000, KDD '00.

[18]  J A Swets,et al.  Better decisions through science. , 2000, Scientific American.

[19]  Tom Fawcett,et al.  Using rule sets to maximize ROC performance , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[20]  Bianca Zadrozny,et al.  Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers , 2001, ICML.

[21]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[22]  George Forman.  A Method for Discovering the Insignificance of One's Best Classifier and the Unlearnability of a Classification Task , 2002 .

[23]  Sofus A. Macskassy,et al.  Confidence Bands for Roc Curves , 2003 .

[24]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[25]  Filippo Neri,et al.  Learning in the “Real World” , 1998, Machine Learning.

[26]  Stan Matwin,et al.  Machine Learning for the Detection of Oil Spills in Satellite Radar Images , 1998, Machine Learning.

[27]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[28]  Tom Fawcett,et al.  Adaptive Fraud Detection , 1997, Data Mining and Knowledge Discovery.

[29]  R. .. Roberts.  Repairing Concavities in ROC Curves , 2004 .