A review of statistical updating methods for clinical prediction models

A clinical prediction model is a tool for predicting healthcare outcomes, usually within a specific population and context. A common approach is to develop a new clinical prediction model for each population and context; however, this wastes potentially useful historical information. A better approach is to update or incorporate the existing clinical prediction models already developed for use in similar contexts or populations. In addition, clinical prediction models commonly become miscalibrated over time, and need replacing or updating. In this article, we review a range of approaches for re-using and updating clinical prediction models; these fall in into three main categories: simple coefficient updating, combining multiple previous clinical prediction models in a meta-model and dynamic updating of models. We evaluated the performance (discrimination and calibration) of the different strategies using data on mortality following cardiac surgery in the United Kingdom: We found that no single strategy performed sufficiently well to be used to the exclusion of the others. In conclusion, useful tools exist for updating existing clinical prediction models to a new population or context, and these should be implemented rather than developing a new clinical prediction model from scratch, using a breadth of complementary statistical methods.

[1]  S. Nashef,et al.  The logistic EuroSCORE , 2003 .

[2]  Ewout W. Steyerberg,et al.  Improving patient prostate cancer risk assessment: Moving from static, globally-applied to dynamic, practice-specific risk calculators , 2015, J. Biomed. Informatics.

[3]  Sabine Van Huffel,et al.  Simple dichotomous updating methods improved the validity of polytomous prediction models. , 2013, Journal of clinical epidemiology.

[4]  Stuart W Grant,et al.  Clinical registries: governance, management, analysis and applications. , 2013, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[5]  H C Van Houwelingen,et al.  Construction, validation and updating of a prognostic model for kidney graft survival. , 1995, Statistics in medicine.

[6]  Iain Buchan,et al.  Dynamic Prediction Modeling Approaches for Cardiac Surgery , 2013, Circulation. Cardiovascular quality and outcomes.

[7]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: validating a prognostic model , 2009, BMJ : British Medical Journal.

[8]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[9]  Y Vergouwe,et al.  Updating methods improved the performance of a clinical prediction model in new patients. , 2008, Journal of clinical epidemiology.

[10]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3--valve plus coronary artery bypass grafting surgery. , 2009, The Annals of thoracic surgery.

[11]  Yvonne Vergouwe,et al.  A simple method to adjust clinical prediction models to local circumstances , 2009, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[12]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[13]  G. Bonsel,et al.  The Risk of Severe Postoperative Pain: Modification and Validation of a Clinical Prediction Rule , 2008, Anesthesia and analgesia.

[14]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[15]  P. Austin,et al.  Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models , 2014, Statistical methods in medical research.

[16]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery. , 2009, The Annals of thoracic surgery.

[17]  Iain Buchan,et al.  Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. , 2013, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[18]  Richard D Riley,et al.  Prognosis research strategy (PROGRESS) 4: Stratified medicine research , 2013, BMJ : British Medical Journal.

[19]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[20]  Samer A M Nashef,et al.  EuroSCORE II. , 2012, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[21]  Gengsheng Qin,et al.  Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test , 2008, Statistical methods in medical research.

[22]  Peter Dalgaard,et al.  R Development Core Team (2010): R: A language and environment for statistical computing , 2010 .

[23]  Karel G M Moons,et al.  A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta‐analysis , 2013, Statistics in medicine.

[24]  S. Lemeshow,et al.  European system for cardiac operative risk evaluation (EuroSCORE). , 1999, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[25]  David Madigan,et al.  Dynamic Logistic Regression and Dynamic Model Averaging for Binary Classification , 2012, Biometrics.

[26]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[27]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery. , 2009, The Annals of thoracic surgery.

[28]  D. Rubin,et al.  Statistical Analysis with Missing Data , 1988 .

[29]  K. Covinsky,et al.  Assessing the Generalizability of Prognostic Information , 1999, Annals of Internal Medicine.

[30]  Karel G M Moons,et al.  Meta‐analysis and aggregation of multiple published prediction models , 2014, Statistics in medicine.

[31]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[32]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[33]  A. Gelman,et al.  Multiple Imputation with Diagnostics (mi) in R: Opening Windows into the Black Box , 2011 .

[34]  Yvonne Vergouwe,et al.  Adaptation of Clinical Prediction Models for Application in Local Settings , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[35]  Ewout W. Steyerberg,et al.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers , 2013, Statistics in medicine.

[36]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[37]  Ray Moynihan Court hears how drug giant Merck tried to “neutralise” and “discredit” doctors critical of Vioxx , 2009, BMJ : British Medical Journal.

[38]  Karel G M Moons,et al.  Aggregating published prediction models with individual participant data: a comparison of different approaches , 2012, Statistics in medicine.

[39]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[40]  A. Bryan,et al.  How does EuroSCORE II perform in UK cardiac surgery; an analysis of 23 740 patients from the Society for Cardiothoracic Surgery in Great Britain and Ireland National Database , 2012, Heart.

[41]  Stanley Lemeshow,et al.  Multiple Logistic Regression , 2005 .

[42]  H C van Houwelingen,et al.  Validation, calibration, revision and combination of prognostic survival models. , 2000, Statistics in medicine.

[43]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 2: Prognostic Factor Research , 2013, PLoS medicine.

[44]  Xavier Robin,et al.  pROC: an open-source package for R and S+ to analyze and compare ROC curves , 2011, BMC Bioinformatics.

[45]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[46]  G. Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement , 2015, Annals of Internal Medicine.

[47]  Mehryar Mohri,et al.  Confidence Intervals for the Area Under the ROC Curve , 2004, NIPS.

[48]  Hans C. van Houwelingen,et al.  Validation, calibration, revision and combination of prognostic survival models , 2000 .

[49]  M. Caputo,et al.  Control chart methods for monitoring cardiac surgical performance and their interpretation. , 2004, The Journal of thoracic and cardiovascular surgery.

[50]  G. Bedogni,et al.  Clinical Prediction Models—a Practical Approach to Development, Validation and Updating , 2009 .

[51]  E. Blackstone,et al.  Using Society of Thoracic Surgeons risk models for risk-adjusting cardiac surgery results. , 2010, The Annals of thoracic surgery.

[52]  D. Cox Two further applications of a model for binary regression , 1958 .

[53]  Chris Sherlaw-Johnson,et al.  Monitoring the results of cardiac surgery by variable life-adjusted display , 1997, The Lancet.

[54]  Adrian E. Raftery,et al.  Prediction under Model Uncertainty Via Dynamic Model Averaging : Application to a Cold Rolling Mill 1 , 2008 .

[55]  Richard D Riley,et al.  Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes , 2013, BMJ : British Medical Journal.

[56]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data: Little/Statistical Analysis with Missing Data , 2002 .

[57]  P. Beattie,et al.  Clinical prediction rules: what are they and what do they tell us? , 2006, The Australian journal of physiotherapy.