Confidence intervals around Bayes Cost in multi‐state diagnostic settings to estimate optimal performance

A critical feature of diagnostic testing is correctly classifying subjects based upon specified thresholds of some measure. The commonly employed Youden index determines a test's optimal thresholds by maximizing the correct classification rates for a diagnostic scenario. An alternative to the Youden index is the cost function, Bayes Cost (BC). BC determines a test's optimal setting by minimizing the sum of all misclassification rates from the test. Unlike the Youden index, BC can consider a priori costs of all the diagnostic outcomes including class specific misclassifications regardless of the number of classes. Delta method approximate confidence intervals around BC are derived under the assumption of normally distributed classes as a means for quantifying a test's performance and comparing classifiers at their optimal settings in a multi-state diagnostic framework. A simulation study is conducted to demonstrate the performance of the derived confidence intervals that are found to perform well, especially for sample sizes of 50 or larger in each diagnostic class. Finally, the proposed methods are applied to a four-class breast tissue classification problem, where four possible discriminatory features are compared under varying decision cost structures. Using the confidence intervals around BC, the best feature for classification is selected, and the optimal thresholds and their 95% confidence intervals are determined.

[1]  Lili Tian,et al.  Exact confidence interval estimation for the Youden index and its corresponding optimal cut-point , 2012, Comput. Stat. Data Anal..

[2]  Steven N. Thorsen,et al.  The ROC manifold for classification systems , 2011, Pattern Recognit..

[3]  Constantin T Yiannoutsos,et al.  Accuracy and cut‐off point selection in three‐class classification problems using a generalization of the Youden index , 2010, Statistics in medicine.

[4]  Enrique F. Schisterman,et al.  Confidence Intervals for the Youden Index and Corresponding Optimal Cut-Point , 2007, Commun. Stat. Simul. Comput..

[5]  Lluís Jover,et al.  Optimum threshold estimation based on cost function in a multistate diagnostic setting , 2012, Statistics in medicine.

[6]  Enrique F Schisterman,et al.  Youden Index and the optimal threshold for markers with mass at zero , 2008, Statistics in medicine.

[7]  Muni S. Srivastava,et al.  Regression Analysis: Theory, Methods, and Applications , 1991 .

[8]  J. Jossinet,et al.  Classification of breast tissue by electrical impedance spectroscopy , 2006, Medical and Biological Engineering and Computing.

[9]  Brani Vidakovic,et al.  Nonparametric Statistics with Applications to Science and Engineering (Wiley Series in Probability and Statistics) , 2007 .

[10]  Martine Wallon,et al.  Methods to Estimate the Optimal Threshold for Normally or Log-Normally Distributed Biological Tests , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[11]  Lluís Jover,et al.  Estimation of the diagnostic threshold accounting for decision costs and sampling uncertainty , 2010, Biometrical journal. Biometrische Zeitschrift.

[12]  Niall M. Adams,et al.  Comparing classifiers when the misallocation costs are uncertain , 1999, Pattern Recognit..

[13]  Jingqin Luo,et al.  Youden Index and Associated Cut-Points for Three Ordinal Diagnostic Groups , 2013, Commun. Stat. Simul. Comput..

[14]  Todd A Alonzo,et al.  Generalization of Youden index for multiple‐class classification problems applied to the assessment of externally validated cognition in Parkinson disease screening , 2013, Statistics in medicine.

[15]  Charles E. Metz,et al.  Optimization of Restricted ROC Surfaces in Three-Class Classification Tasks , 2007, IEEE Transactions on Medical Imaging.

[16]  T. Utlaut.  Nonparametric Statistics with Applications to Science and Engineering , 2008 .

[17]  D. McClish.  Evaluation of the accuracy of medical tests in a region around the optimal point. , 2012, Academic radiology.

[18]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[19]  Amos Gilat,et al.  Numerical Methods for Engineers and Scientists: An Introduction with Applications Using MATLAB , 2007 .

[20]  N. Perkins,et al.  The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. , 2006, American journal of epidemiology.

[21]  B. Reiser,et al.  Estimation of the Youden Index and its Associated Cutoff Point , 2005, Biometrical journal. Biometrische Zeitschrift.

[22]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[23]  Xin He,et al.  Three-Class ROC Analysis—The Equal Error Utility Assumption and the Optimality of Three-Class ROC Surface Using the Ideal Observer , 2006, IEEE Transactions on Medical Imaging.