Relationship between Brier score and area under the binormal ROC curve

If we consider the Brier score (B) in the context of the signal detection theory and assume that it makes sense to consider the existence of B as a parameter for the population (let B be this B), and if we assume that the calibration in the observer's probability estimate is perfect, we find that there is a theoretical relationship between B and the area under the binormal receiver operating characteristic (ROC) curve, A(Z). We have derived this theoretical functional relationship between B and A(Z), by using the parameter a and b in the binormal ROC model and the prior probability of signal events (alpha); here, the two underlying normal distributions are N and N; and, a= and b=. We empirically found that, if parameters b and alpha are constant, B values in relation to given A(Z) values monotonically decrease as A(Z) values increase, and these relationship curves have monotonically decreasing slopes.

[1]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[2]  R M Centor,et al.  Eualuating Physicians' Probabilistic Judgments , 1988, Medical decision making : an international journal of the Society for Medical Decision Making.

[3]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[4]  D. Bamber The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[5]  C. Metz,et al.  Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. , 1998, Statistics in medicine.

[6]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[7]  E Somoza,et al.  Classifying Binormal Diagnostic Tests Using Separation - Asymmetry Diagrams with Constant-performance Curves , 1994, Medical decision making : an international journal of the Society for Medical Decision Making.

[8]  D J Spiegelhalter,et al.  Probabilistic prediction in patient management and clinical trials. , 1986, Statistics in medicine.

[9]  D. McClish,et al.  How Well Can Physicians Estimate Mortality in a Medical Intensive Care Unit? , 1989, Medical decision making : an international journal of the Society for Medical Decision Making.

[10]  D A Redelmeier,et al.  Assessing predictive accuracy: how to compare Brier scores. , 1991, Journal of clinical epidemiology.

[11]  J. Gurney,et al.  Solitary pulmonary nodules: determining the likelihood of malignancy with neural network analysis. , 1995, Radiology.

[12]  A. Simpson,et al.  What is the best index of detectability? , 1973, Psychological Bulletin.

[13]  John A. Swets,et al.  Evaluation of diagnostic systems : methods from signal detection theory , 1982 .

[14]  D Mossman,et al.  Resampling Techniques in the Analysis of Non-binormal ROC Data , 1995, Medical decision making : an international journal of the Society for Medical Decision Making.

[15]  James P. Egan,et al.  Signal detection theory and ROC analysis , 1975 .

[16]  J W Gurney Neural networks at the crossroads: caution ahead. , 1994, Radiology.