ROC curves for regression

Receiver Operating Characteristic (ROC) analysis is one of the most popular tools for the visual assessment and understanding of classifier performance. In this paper we present a new representation of regression models in the so-called regression ROC (RROC) space. The basic idea is to represent over-estimation against under-estimation. The curves are just drawn by adjusting a shift, a constant that is added (or subtracted) to the predictions, and plays a similar role as a threshold in classification. From here, we develop the notions of optimal operating condition, convexity, dominance, and explore several evaluation metrics that can be shown graphically, such as the area over the RROC curve (AOC). In particular, we show a novel and significant result: the AOC is equivalent to the error variance. We illustrate the application of RROC curves to resource estimation, namely the estimation of software project effort.

[1]  F. Diebold,et al.  Further results on forecasting and model selection under asymmetric loss , 1996 .

[2]  Peter A. Flach.  The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics , 2003, ICML.

[3]  Sven F. Crone,et al.  Training artificial neural networks for time series prediction using asymmetric cost functions , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[4]  Robert C. Holte,et al.  Cost curves: An improved method for visualizing classifier performance , 2006, Machine Learning.

[5]  Andrew Beng Jin Teoh,et al.  An online AUC formulation for binary classification , 2012, Pattern Recognit..

[6]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[7]  FlachPeter,et al.  A unified view of performance metrics , 2012 .

[8]  Isabella Wieczorek,et al.  Resource Estimation in Software Engineering , 2002 .

[9]  Steven N. Thorsen,et al.  The ROC manifold for classification systems , 2011, Pattern Recognit..

[10]  Robert Sabourin,et al.  Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs , 2010, Pattern Recognit..

[11]  Luís Torgo,et al.  Regression error characteristic surfaces , 2005, KDD '05.

[12]  Christian Janssen,et al.  Real estate price prediction under asymmetric loss , 1995 .

[13]  Peter A. Flach,et al.  A Coherent Interpretation of AUC as a Measure of Aggregated Classification Performance , 2011, ICML.

[14]  José Hernández-Orallo,et al.  An experimental comparison of performance measures for classification , 2009, Pattern Recognit. Lett..

[15]  Mark Anthony Clatworthy,et al.  Are Analysts’ Loss Functions Asymmetric? , 2012 .

[16]  Mario Jino,et al.  Machine Learning Methods and Asymmetric Cost Function to Estimate Execution Effort of Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[17]  Robert Sabourin,et al.  Boolean Combination of Classifiers in the ROC Space , 2010, 2010 20th International Conference on Pattern Recognition.

[18]  Peter A. Flach,et al.  The 1st workshop on ROC analysis in artificial intelligence (ROCAI-2004) , 2004, SKDD.

[19]  Matei Demetrescu,et al.  An extension of the Gauss-Newton algorithm for estimation under asymmetric loss , 2006, Comput. Stat. Data Anal..

[20]  Huimin Zhao,et al.  Tuning Data Mining Methods for Cost-Sensitive Regression: A Study in Loan Charge-Off Forecasting , 2008, J. Manag. Inf. Syst..

[21]  David J. Hand,et al.  ROC Curves for Continuous Data , 2009 .

[22]  Peter D. Turney.  Types of Cost in Inductive Concept Learning , 2002, ArXiv.

[23]  C. Marroccoa,et al.  Maximizing the area under the ROC curve by pairwise feature combination , 2008 .

[24]  Asit P. Basu,et al.  Asymmetric Loss Functions for Estimating System Reliability , 1996 .

[25]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[26]  Nader Ebrahimi,et al.  Bayesian approach to life testing and reliability estimation using asymmetric loss function , 1991 .

[27]  Huimin Zhao,et al.  An extended tuning method for cost-sensitive regression and forecasting , 2011, Decis. Support Syst..

[28]  Hiroshi Mamitsuka,et al.  Selecting features in microarray classification using ROC curves , 2006, Pattern Recognit..

[29]  Lucila Ohno-Machado,et al.  The use of receiver operating characteristic curves in biomedical informatics , 2005, J. Biomed. Informatics.

[30]  F. Diebold,et al.  Optimal Prediction Under Asymmetric Loss , 1994, Econometric Theory.

[31]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[32]  Andrew J. Patton,et al.  Testing Forecast Optimality Under Unknown Loss , 2007 .

[33]  J A Swets,et al.  Better decisions through science. , 2000, Scientific American.

[34]  Clive W. J. Granger,et al.  Prediction with a generalized cost of error function , 1969 .

[35]  Francesco Tortorella,et al.  Partial AUC maximization in a linear combination of dichotomizers , 2011, Pattern Recognit..

[36]  A. Zellner.  Bayesian Estimation and Prediction Using Asymmetric Loss Functions , 1986 .

[37]  Peter A. Flach,et al.  Decision Support for Data Mining , 2003 .

[38]  Peter A. Flach,et al.  Learning Decision Trees Using the Area Under the ROC Curve , 2002, ICML.

[39]  David J. Hand,et al.  Measuring classifier performance: a coherent alternative to the area under the ROC curve , 2009, Machine Learning.

[40]  José Hernández-Orallo,et al.  Volume under the ROC Surface for Multi-class Problems , 2003, ECML.

[41]  Robert Sabourin,et al.  Adaptive ROC-based ensembles of HMMs applied to anomaly detection , 2012, Pattern Recognit..

[42]  Shari Lawrence Pfleeger,et al.  An empirical study of maintenance and development estimation accuracy , 2002, J. Syst. Softw..

[43]  James E. Goin,et al.  ROC curve estimation and hypothesis testing: applications to breast cancer detection , 1982, Pattern Recognit..

[44]  Peter A. Flach,et al.  Brier Curves: a New Cost-Based Visualisation of Classifier Performance , 2011, ICML.

[45]  Peter A. Flach,et al.  ROC curves in cost space , 2013, Machine Learning.

[46]  Clive W. J. Granger,et al.  Outline of forecast theory using generalized cost functions , 1999 .

[47]  Kar-Ann Toh,et al.  Maximizing area under ROC curve for biometric scores fusion , 2008, Pattern Recognit..

[48]  Luís Torgo,et al.  Utility-Based Regression , 2007, PKDD.

[49]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[50]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[51]  L B Lusted,et al.  Signal detectability and medical decision-making. , 1971, Science.

[52]  Bianca Zadrozny,et al.  Ranking-based evaluation of regression models , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[53]  Jinbo Bi,et al.  Regression Error Characteristic Curves , 2003, ICML.

[54]  Peter A. Flach,et al.  A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss C` Esar Ferri , 2012 .