Technical Note: Towards ROC Curves in Cost Space

AbstractROC curves and cost curves are two popular ways of visualising classifier performance, finding appro-priate thresholds according to the operating condition, and deriving useful aggregated measures such asthe area under the ROC curve (AUC) or the area under the optimal cost curve. In this note we presentsome new findings and connections between ROC space and cost space, by using the expected loss overa range of operating conditions. In particular, we show that ROC curves can be transferred to cost spaceby means of a very natural way of understanding how thresholds should be chosen, by selecting thethreshold such that the proportion of positive predictions equals the operating condition (either in theform of cost proportion or skew). We call these new curves ROC cost curves, and we demonstrate thatthe expected loss as measured by the area under these curves is linearly related to AUC. This opens upa series of new possibilities and clarifies the notion of cost curve and its relation to ROC analysis. Inaddition, we show that for a classifier that assigns the scores in an evenly-spaced way, these curves areequal to the Brier curves. As a result, this establishes the first clear connection between AUC and theBrier score.Keywords: cost curves, ROC curves, Brier curves, classifier performance measures, cost-sensitive eval-uation, operating condition, Brier score, Area Under the ROC Curve (AUC).