Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models

Net reclassification and integrated discrimination improvements have been proposed as alternatives to the increase in the area under the curve for evaluating improvement in the performance of risk assessment algorithms introduced by the addition of new phenotypic or genetic markers. In this paper, we demonstrate that in the setting of linear discriminant analysis, under the assumptions of multivariate normality, all three measures can be presented as functions of the squared Mahalanobis distance. This relationship affords an interpretation of the magnitude of these measures in the familiar language of effect size for uncorrelated variables. Furthermore, it allows us to conclude that net reclassification improvement can be viewed as a universal measure of effect size. Our theoretical developments are illustrated with an example based on the Framingham Heart Study risk assessment model for high-risk men in primary prevention of cardiovascular disease.

[1]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[2]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[3]  M. Pepe,et al.  Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. , 2004, American journal of epidemiology.

[4]  N. Cook.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[5]  Nancy R. Cook,et al.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[6]  Ewout W Steyerberg,et al.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers , 2011, Statistics in medicine.

[7]  M. Pencina,et al.  On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data , 2011, Statistics in medicine.

[8]  D.,et al.  Regression Models and Life-Tables , 2022 .

[9]  J. Ware.  The limitations of risk factors as prognostic tools. , 2006, The New England journal of medicine.

[10]  D. F. Morrison,et al.  Multivariate Statistical Methods , 1968 .

[11]  David J Hand,et al.  Evaluating diagnostic tests: The area under the ROC curve and the balance of errors , 2010, Statistics in medicine.

[12]  P. Heagerty,et al.  Survival Model Predictive Accuracy and ROC Curves , 2005, Biometrics.

[13]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[14]  M. Pencina,et al.  General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study , 2008, Circulation.

[15]  Holly Janes,et al.  Assessing the Value of Risk Predictions by Using Risk Stratification Tables , 2008, Annals of Internal Medicine.

[16]  Jun S. Liu,et al.  Linear Combinations of Multiple Diagnostic Markers , 1993 .

[17]  J. Ioannidis,et al.  Assessment of claims of improved prediction beyond the Framingham risk score. , 2009, JAMA.

[18]  E. Elkin,et al.  Decision Curve Analysis: A Novel Method for Evaluating Prediction Models , 2006, Medical decision making : an international journal of the Society for Medical Decision Making.

[19]  Lloyd E Chambless,et al.  Several methods to assess improvement in risk prediction models: Extension to survival analysis , 2011, Statistics in medicine.

[20]  M S Pepe,et al.  Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929) , 2008, Statistics in medicine.

[21]  Michael J. Pencina,et al.  Comments on ‘Integrated discrimination and net reclassification improvements—Practical advice’ , 2008 .

[22]  J. Kastelein,et al.  C-reactive protein and cardiovascular risk: more fuel to the fire , 2010, The Lancet.

[23]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[24]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[25]  Daniel B. Mark,et al.  TUTORIAL IN BIOSTATISTICS MULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS , 1996 .

[26]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[27]  Mari Palta,et al.  Properties of R2 statistics for logistic regression , 2006, Statistics in medicine.

[28]  M. Gail.  Value of adding single-nucleotide polymorphism genotypes to a breast cancer risk model. , 2009, Journal of the National Cancer Institute.

[29]  Strother H. Walker,et al.  Estimation of the probability of an event as a function of several independent variables. , 1967, Biometrika.

[30]  Tue Tjur,et al.  Coefficients of Determination in Logistic Regression Models—A New Proposal: The Coefficient of Discrimination , 2009 .

[31]  Nancy R Cook,et al.  Using relative utility curves to evaluate risk prediction , 2009, Journal of the Royal Statistical Society. Series A,.

[32]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[33]  M. Pencina,et al.  Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation , 2004, Statistics in medicine.

[34]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[35]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[36]  Michael J Pencina,et al.  Equivalence of improvement in area under ROC curve and linear discriminant analysis coefficient under assumption of normality , 2011, Statistics in medicine.

[37]  Guoqing Diao,et al.  Estimation of time‐dependent area under the ROC curve for long‐term risk prediction , 2006, Statistics in medicine.