Clinical prediction in defined populations: a simulation study investigating when and how to aggregate existing models

BackgroundClinical prediction models (CPMs) are increasingly deployed to support healthcare decisions but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.MethodsSimulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.ResultsWhile redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.ConclusionThis study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

[1]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[2]  P. Austin,et al.  Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models , 2014, Statistical methods in medical research.

[3]  L. Breiman.  Stacked Regressions , 1996, Machine Learning.

[4]  Y Vergouwe,et al.  Updating methods improved the performance of a clinical prediction model in new patients. , 2008, Journal of clinical epidemiology.

[5]  J. Habbema,et al.  Prognostic Modeling with Logistic Regression Analysis , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[6]  M. Henmi,et al.  Synthesis of clinical prediction models under different sets of covariates with one individual patient data , 2015, BMC Medical Research Methodology.

[7]  Karel G M Moons,et al.  Meta‐analysis and aggregation of multiple published prediction models , 2014, Statistics in medicine.

[8]  Xavier Basagaña,et al.  Methods for Handling Missing Variables in Risk Prediction Models. , 2016, American journal of epidemiology.

[9]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery. , 2009, The Annals of thoracic surgery.

[10]  Xavier Robin,et al.  pROC: an open-source package for R and S+ to analyze and compare ROC curves , 2011, BMC Bioinformatics.

[11]  G. Collins,et al.  Prediction models for cardiovascular disease risk in the general population: systematic review , 2016, British Medical Journal.

[12]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[13]  D. Cox.  Two further applications of a model for binary regression , 1958 .

[14]  Michael J. Pazzani,et al.  A Principal Components Approach to Combining Regression Estimates , 1999, Machine Learning.

[15]  Karel G M Moons,et al.  Aggregating published prediction models with individual participant data: a comparison of different approaches , 2012, Statistics in medicine.

[16]  R. D. Alley,et al.  The society of thoracic surgeons. , 1976, The Annals of thoracic surgery.

[17]  Richard D Riley,et al.  Ten steps towards improving prognosis research , 2009, BMJ : British Medical Journal.

[18]  Y. Vergouwe,et al.  Validation, updating and impact of clinical prediction rules: a review. , 2008, Journal of clinical epidemiology.

[19]  Andreas Beckmann,et al.  German Aortic Valve Score: a new scoring system for prediction of mortality related to aortic valve procedures in adults. , 2013, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[20]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery. , 2009, The Annals of thoracic surgery.

[21]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[22]  E W Steyerberg,et al.  See Blockindiscussions, Blockinstats, Blockinand Blockinauthor Blockinprofiles Blockinfor Blockinthis Blockinpublication Prognostic Blockinmodels Blockinbased Blockinon Blockinliterature Blockinand Individual Blockinpatient Blockindata Blockinin Blockinlogistic Blockinregression Analysis Article Blo , 2022 .

[23]  Samer A M Nashef,et al.  EuroSCORE II. , 2012, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[24]  Karel G M Moons,et al.  A new framework to enhance the interpretation of external validation studies of clinical prediction models. , 2015, Journal of clinical epidemiology.

[25]  Richard D Riley,et al.  Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. , 2007, Journal of clinical epidemiology.

[26]  Thomas Jaki,et al.  Recovering Independent Associations in Genetics: A Comparison , 2012, J. Comput. Biol..

[27]  S. Nashef,et al.  The logistic EuroSCORE , 2003 .

[28]  Thomas Jaki,et al.  A review of statistical updating methods for clinical prediction models , 2018, Statistical methods in medical research.

[29]  Douglas G Altman,et al.  Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer , 2009, Cancer investigation.

[30]  M. Maumy-Bertrand,et al.  plsRglm: Partial least squares linear and generalized linear regression for processing incomplete datasets by cross-validation and bootstrap techniques with R , 2018, 1810.01005.

[31]  N F de Keizer,et al.  External validation of prognostic models for critically ill patients required substantial sample sizes. , 2007, Journal of clinical epidemiology.

[32]  Gary S Collins,et al.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study , 2015, Statistics in medicine.

[33]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: Developing a prognostic model , 2009, BMJ : British Medical Journal.

[34]  P. Royston,et al.  Prognosis and prognostic research: application and impact of prognostic models in clinical practice , 2009, BMJ : British Medical Journal.

[35]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: validating a prognostic model , 2009, BMJ : British Medical Journal.

[36]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[37]  E W Steyerberg,et al.  Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. , 1999, Journal of clinical epidemiology.

[38]  Yvonne Vergouwe,et al.  Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. , 2005, Journal of clinical epidemiology.

[39]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[40]  Sean M. O'Brien,et al.  The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3--valve plus coronary artery bypass grafting surgery. , 2009, The Annals of thoracic surgery.

[41]  Yvonne Vergouwe,et al.  Adaptation of Clinical Prediction Models for Application in Local Settings , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.