Temporal recalibration for improving prognostic model development and risk predictions in settings where survival is improving over time

Abstract Background Prognostic models are typically developed in studies covering long time periods. However, if more recent years have seen improvements in survival, then using the full dataset may lead to out-of-date survival predictions. Period analysis addresses this by developing the model in a subset of the data from a recent time window, but results in a reduction of sample size. Methods We propose a new approach, called temporal recalibration, to combine the advantages of period analysis and full cohort analysis. This approach develops a model in the entire dataset and then recalibrates the baseline survival using a period analysis sample. The approaches are demonstrated utilizing a prognostic model in colon cancer built using both Cox proportional hazards and flexible parametric survival models with data from 1996–2005 from the Surveillance, Epidemiology, and End Results (SEER) Program database. Comparison of model predictions with observed survival estimates were made for new patients subsequently diagnosed in 2006 and followed-up until 2015. Results Period analysis and temporal recalibration provided more up-to-date survival predictions that more closely matched observed survival in subsequent data than the standard full cohort models. In addition, temporal recalibration provided more precise estimates of predictor effects. Conclusion Prognostic models are typically developed using a full cohort analysis that can result in out-of-date long-term survival estimates when survival has improved in recent years. Temporal recalibration is a simple method to address this, which can be used when developing and updating prognostic models to ensure survival predictions are more closely calibrated with the observed survival of individuals diagnosed subsequently.

[1]  H. Weir,et al.  Cervical cancer survival in the United States by race and stage (2001‐2009): Findings from the CONCORD‐2 study , 2017, Cancer.

[2]  Paul C Lambert,et al.  Providing more up-to-date estimates of patient survival: a comparison of standard survival analysis with period analysis using life-table methods and proportional hazards models. , 2004, Journal of clinical epidemiology.

[3]  P. Royston,et al.  External validation of a Cox prognostic model: principles and methods , 2013, BMC Medical Research Methodology.

[4]  R. Szczesniak,et al.  Up-to-date and projected estimates of survival for people with cystic fibrosis using baseline characteristics: A longitudinal study using UK patient registry data , 2018, Journal of cystic fibrosis : official journal of the European Cystic Fibrosis Society.

[5]  Patrick Royston.  STCSTAT2: Stata module to compute Harrell's c-index for flexible parametric models , 2011 .

[6]  Paul C. Lambert,et al.  Comparison of different approaches to estimating age standardized net survival , 2015, BMC Medical Research Methodology.

[7]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[8]  Paul C. Lambert,et al.  Further Development of Flexible Parametric Models for Survival Analysis , 2009 .

[9]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[10]  S. le Cessie,et al.  Predictive value of statistical models. , 1990, Statistics in medicine.

[11]  Richard D Riley,et al.  Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? , 2017, Statistical methods in medical research.

[12]  S Siesling,et al.  Improved survival of colon cancer due to improved treatment and detection: a nationwide population-based study in The Netherlands 1989-2006. , 2010, Annals of oncology : official journal of the European Society for Medical Oncology.

[13]  D.,et al.  Regression Models and Life-Tables , 2022 .

[14]  Carol Coupland,et al.  Development and validation of risk prediction equations to estimate survival in patients with colorectal cancer: cohort study , 2017, British Medical Journal.

[15]  B Rachet,et al.  Cancer survival in Australia, Canada, Denmark, Norway, Sweden, and the UK, 1995–2007 (the International Cancer Benchmarking Partnership): an analysis of population-based cancer registry data , 2011, Lancet.

[16]  D. Mark,et al.  Clinical prediction models: are we building better mousetraps? , 2003, Journal of the American College of Cardiology.

[17]  H. Weir,et al.  Liver cancer survival in the United States by race and stage (2001‐2009): Findings from the CONCORD‐2 study , 2017, Cancer.

[18]  Patrick Royston,et al.  Reporting performance of prognostic models in cancer: a review , 2010, BMC medicine.

[19]  E. Steyerberg,et al.  Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings , 2008, Epidemiology and Infection.

[20]  Frank E. Harrell,et al.  Prediction models need appropriate internal, internal-external, and external validation. , 2016, Journal of clinical epidemiology.

[21]  Ewout W Steyerberg,et al.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests , 2016, British Medical Journal.

[22]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[23]  H Brenner,et al.  Higher long-term cancer survival rates in southeastern Netherlands using up-to-date period analysis. , 2006, Annals of oncology : official journal of the European Society for Medical Oncology.

[24]  Ian O. Ellis,et al.  An updated PREDICT breast cancer prognostication and treatment benefit prediction model with independent validation , 2017, Breast Cancer Research.

[25]  Á. Theodórs,et al.  Why is colon cancer survival improving by time? A nationwide survival analysis spanning 35 years , 2017, International journal of cancer.

[26]  Martha Sajatovic,et al.  Clinical Prediction Models , 2013 .

[27]  Andrew Miles.  Obtaining Predictions from Models Fit to Multiply Imputed Data , 2016 .

[28]  P. Lambert,et al.  Robustness of individual and marginal model-based estimates: A sensitivity analysis of flexible parametric models , 2019, Cancer epidemiology.

[29]  Paul C Lambert,et al.  Flexible parametric modelling of cause-specific hazards to estimate cumulative incidence functions , 2013, BMC Medical Research Methodology.

[30]  Helena Carreira,et al.  Global surveillance of cancer survival 1995–2009: analysis of individual data for 25 676 887 patients from 279 population-based registries in 67 countries (CONCORD-2) , 2015, The Lancet.

[31]  M. Rosén,et al.  Cancer patient survival in Sweden at the beginning of the third millennium – predictions using period analysis , 2004, Cancer Causes & Control.

[32]  Paul C. Lambert,et al.  Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model , 2011 .

[33]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[34]  P. Royston,et al.  Flexible parametric proportional‐hazards and proportional‐odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects , 2002, Statistics in medicine.

[35]  M. Coleman,et al.  Colon cancer survival in the United States by race and stage (2001‐2009): Findings from the CONCORD‐2 study , 2017, Cancer.

[36]  L. Ellison,et al.  An empirical evaluation of period survival analysis using data from the Canadian Cancer Registry. , 2006, Annals of epidemiology.

[37]  Richard D Riley,et al.  Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes , 2018, Statistics in medicine.

[38]  Hermann Brenner,et al.  Use of period analysis for providing more up-to-date estimates of long-term survival rates: empirical evaluation among 370,000 cancer patients in Finland. , 2002, International journal of epidemiology.

[39]  M. Chávez-MacGregor,et al.  Cancer survival in Australia, Canada, Denmark, Norway, Sweden, and the UK, 1995-2007 (the International Cancer Benchmarking Partnership): An analysis of population-based cancer registry data , 2011 .