Roc Confidence Bands: An Empirical Study

This paper is about constructing confidence bands around an ROCcurve such that (1 - \delta)% of the ROC curves traced by data setsof size r will fall completely within the bands. We introduce tothe machine learning community three methods from the medicalfield that are applicable to generate such bands. We then evaluatethese methods on the simple case of A'Â'A ¢A' A ¬A' A ÂSbinormalA'Â'A ¢A' A ¬A' A Â? distributionsA'Â'A ¢A' A ¬A' A  the scores for positive and the score for negative instances aredrawn from two normal distributions. We show that none of themethods generate appropriate bands and investigate two types ofvariances problems. We show that widening the bands does notproduce the proper bandwidths but that fitting a normal distributionto the observed drawn samples and drawing samples fromthis distribution (parametric bootstrap) does generate bands thatare much closer to the desired coverage although still not perfect.We tested the original methods as well as parametric bootstrap onthe covertype data set from the UCI ML-repority. The originalmethods perform the same as in the synthetic case, whereas theparametric bootstrap technique did not yield the expected results.This is primarily due to not being able to generate a good fit forthe score distributions. Whether it is possible to fit well-behavingparametric distribution to learned models is an open question weleave to the machine learning community to answer.