A Bias-Corrected Net Reclassification Improvement for Clinical Subgroups

Background. Comparing prediction models using reclassification within subgroups at intermediate risk is often of clinical interest. Objective. To demonstrate a method for obtaining an unbiased estimate for the Net Reclassification Improvement (NRI) evaluated only on a subset, the clinical NRI. Study Design and Setting. We derived the expected value of the clinical NRI under the null hypothesis using the same principles as the overall NRI. We then conducted a simulation study based on a logistic model with a known predictor and a potential predictor, varying the effects of the known and potential predictors to test the performance of our bias-corrected clinical NRI measure. Finally, data from the Women’s Health Study, a prospective cohort of 24 171 female health professionals, were used as an example of the proposed method. Results. Our bias-corrected estimate is shown to have a mean of zero in the null case under a range of simulated parameters and, unlike the naïve estimate, to be unbiased. We also provide 2 methods for obtaining a variance estimate, both with reasonable type 1 errors. Conclusion. Our proposed method is an improvement over currently used methods of calculating the clinical NRI and is recommended to reduce overly optimistic results.

[1]  I-Min Lee,et al.  A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women , 2005 .

[2]  N. Cook,et al.  Baseline characteristics of participants in the Women's Health Study. , 2000, Journal of women's health & gender-based medicine.

[3]  Sander Greenland,et al.  The need for reorientation toward cost‐effective prediction: Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929) , 2008, Statistics in medicine.

[4]  David A. Hinds,et al.  Assessment of Clinical Validity of a Breast Cancer Risk Model Combining Genetic and Clinical Information , 2010, Journal of the National Cancer Institute.

[5]  E. Boerwinkle,et al.  Impact of Adding a Single Allele in the 9p21 Locus to Traditional Risk Factors on Reclassification of Coronary Heart Disease Risk and Implications for Lipid-Modifying Therapy in the Atherosclerosis Risk in Communities Study , 2009, Circulation. Cardiovascular genetics.

[6]  J. Mckenney,et al.  Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III). , 2001, JAMA.

[7]  Eric Boerwinkle,et al.  Carotid intima-media thickness and presence or absence of plaque improves prediction of coronary heart disease risk: the ARIC (Atherosclerosis Risk In Communities) study. , 2010, Journal of the American College of Cardiology.

[8]  N. Cook,et al.  Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. , 2007, JAMA.

[9]  J. Ware,et al.  Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929) , 2008, Statistics in medicine.

[10]  N. Cook.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[11]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[12]  Sonja Kuhnt,et al.  Bowker's test for symmetry and modifications within the algebraic framework , 2007, Comput. Stat. Data Anal..

[13]  S. Grundy,et al.  National Cholesterol Education Program Third Report of the National Cholesterol Education Program ( NCEP ) Expert Panel on Detection , Evaluation , and Treatment of High Blood Cholesterol in Adults ( Adult Treatment Panel III ) Final Report , 2022 .

[14]  Alice S Whittemore,et al.  Evaluating health risk models , 2009, Statistics in medicine.

[15]  J. Kassirer,et al.  The threshold approach to clinical decision making. , 1980, The New England journal of medicine.

[16]  M. Pencina,et al.  Novel and conventional biomarkers for prediction of incident cardiovascular events in the community. , 2009, JAMA.

[17]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[18]  Kathleen F. Kerr,et al.  Testing for improvement in prediction model performance , 2013, Statistics in medicine.

[19]  L. Peltonen,et al.  A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses , 2010, The Lancet.

[20]  Q. Mcnemar.  Note on the sampling error of the difference between correlated proportions or percentages , 1947, Psychometrika.

[21]  Ewout W Steyerberg,et al.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers , 2011, Statistics in medicine.

[22]  Nancy R Cook,et al.  Performance of reclassification statistics in comparing risk prediction models , 2011, Biometrical journal. Biometrische Zeitschrift.

[23]  F. Harrell,et al.  Criteria for Evaluation of Novel Markers of Cardiovascular Risk: A Scientific Statement From the American Heart Association , 2009, Circulation.

[24]  Lloyd E Chambless,et al.  Several methods to assess improvement in risk prediction models: Extension to survival analysis , 2011, Statistics in medicine.