Multicategory reclassification statistics for assessing improvements in diagnostic accuracy.

In this paper, we extend the definitions of the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) in the context of multicategory classification. Both measures were proposed in Pencina and others (2008. Evaluating the added predictive ability of a new marker: from area under the receiver operating characteristic (ROC) curve to reclassification and beyond. Statistics in Medicine 27, 157-172) as numeric characterizations of accuracy improvement for binary diagnostic tests and were shown to have certain advantage over analyses based on ROC curves or other regression approaches. Estimation and inference procedures for the multiclass NRI and IDI are provided in this paper along with necessary asymptotic distributional results. Simulations are conducted to study the finite-sample properties of the proposed estimators. Two medical examples are considered to illustrate our methodology.

[1]  Michael J Pencina,et al.  Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models , 2012, Statistics in medicine.

[2]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[3]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[4]  M. Pepe,et al.  Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. , 2004, American journal of epidemiology.

[5]  G. Burmester,et al.  Synovitis score: discrimination between chronic low‐grade and high‐grade synovitis , 2006, Histopathology.

[6]  N. Wermuth,et al.  A Comment on the Coefficient of Determination for Binary Responses , 1992 .

[7]  C. Yiannoutsos,et al.  Ordered multiple‐class ROC analysis with continuous measurements , 2004, Statistics in medicine.

[8]  Ewout W Steyerberg,et al.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers , 2011, Statistics in medicine.

[9]  Jialiang Li,et al.  ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. , 2008, Biostatistics.

[10]  M S Pepe,et al.  Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929) , 2008, Statistics in medicine.

[11]  Xiao Song,et al.  Ranking prognosis markers in cancer genomic studies , 2011, Briefings Bioinform..

[12]  S. Menard.  Coefficients of Determination for Multiple Logistic Regression Analysis , 2000 .

[13]  D. Mossman.  Three-way ROCs , 1999, Medical decision making : an international journal of the Society for Medical Decision Making.

[14]  Xiao-Hua Zhou,et al.  Statistical Methods in Diagnostic Medicine , 2002 .

[15]  T. Häupl,et al.  Quantitative determination of the diagnostic accuracy of the synovitis score and its components , 2010, Histopathology.

[16]  Nello Cristianini,et al.  Support vector machine classification and validation of cancer tissue samples using microarray expression data , 2000, Bioinform..

[17]  Tue Tjur,et al.  Coefficients of Determination in Logistic Regression Models—A New Proposal: The Coefficient of Discrimination , 2009 .

[18]  Lucila Ohno-Machado,et al.  An Epicurean learning approach to gene-expression data classification , 2003, Artif. Intell. Medicine.

[19]  C. Gatsonis,et al.  On ROC analysis with nonbinary reference standard , 2012, Biometrical journal. Biometrische Zeitschrift.

[20]  Andreas Alexander Albrecht,et al.  Stochastic local search for the Feature Set problem, with applications to microarray data , 2006, Appl. Math. Comput..

[21]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[22]  P. Hall,et al.  Achieving near perfect classification for functional data , 2012 .

[23]  Yiming Yang,et al.  Analysis of recursive gene selection approaches from microarray data , 2005, Bioinform..

[24]  Jialiang Li,et al.  Weighted area under the receiver operating characteristic curve and its application to gene selection , 2010, Journal of the Royal Statistical Society. Series C, Applied statistics.

[25]  Thomas Lumley,et al.  American Journal of Epidemiology Practice of Epidemiology Evaluating the Incremental Value of New Biomarkers with Integrated Discrimination Improvement , 2022 .