Validation of discrete time‐to‐event prediction models in the presence of competing risks

Abstract Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time‐to‐event outcomes, but the literature on validation methods for discrete time‐to‐event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time‐to‐event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator‐associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.

[1]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[2]  T. Greene,et al.  A simple method to estimate the time-dependent receiver operating characteristic curve and the area under the curve with right censored data , 2018, Statistical methods in medical research.

[3]  Gerhard Tutz,et al.  Modeling Discrete Time-To-Event Data , 2016 .

[4]  Chao Agnes Hsiung,et al.  Counting Process Methods in Survival Analysis , 2014 .

[5]  R B D'Agostino,et al.  Relation of pooled logistic regression to time dependent Cox regression analysis: the Framingham Heart Study. , 1990, Statistics in medicine.

[6]  Sabine Van Huffel,et al.  Assessing calibration of multinomial risk prediction models , 2014, Statistics in medicine.

[7]  Jean-François Dartigues,et al.  Estimating and comparing time‐dependent areas under receiver operating characteristic curves for censored event times with competing risks , 2013, Statistics in medicine.

[8]  Martin Schumacher,et al.  Competing Risks and Multistate Models with R , 2011 .

[9]  Leonhard Held,et al.  Dynamic clinical prediction models for discrete time‐to‐event data with competing risks—A case study on the OUTCOMEREA database , 2018, Biometrical journal. Biometrische Zeitschrift.

[10]  Leonhard Held,et al.  Approximate Bayesian Model Selection with the Deviance Statistic , 2013, 1308.6780.

[11]  Mauricio Valencia,et al.  Ventilator-associated pneumonia , 2009, Current opinion in critical care.

[12]  J. Kalbfleisch,et al.  The Statistical Analysis of Failure Time Data: Kalbfleisch/The Statistical , 2002 .

[13]  B. Souweine,et al.  Ventilator-Associated Events: Prevalence, Outcome, and Relationship With Ventilator-Associated Pneumonia* , 2015, Critical care medicine.

[14]  Laurence L. George,et al.  The Statistical Analysis of Failure Time Data , 2003, Technometrics.

[15]  B. Souweine,et al.  Continuous renal replacement therapy versus intermittent hemodialysis in intensive care patients: impact on mortality and renal recovery , 2016, Intensive Care Medicine.

[16]  Maarten van Smeden,et al.  Sample size considerations and predictive performance of multinomial logistic prediction models , 2019, Statistics in medicine.

[17]  Cécile Proust-Lima,et al.  Quantifying and comparing dynamic predictive accuracy of joint models for longitudinal marker and time‐to‐event in presence of censoring and competing risks , 2015, Biometrics.

[18]  Stijn Vansteelandt,et al.  Attributable mortality of ventilator-associated pneumonia: a reappraisal using causal analysis. , 2011, American journal of respiratory and critical care medicine.

[19]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[20]  M Schumacher,et al.  How to Assess Prognostic Models for Survival Data: A Case Study in Oncology , 2003, Methods of Information in Medicine.

[21]  E Graf,et al.  Assessment and comparison of prognostic classification schemes for survival data. , 1999, Statistics in medicine.

[22]  Gerhard Tutz,et al.  Discrimination measures for discrete time-to-event predictions , 2017 .

[23]  L. Fahrmeir.  Discrete Survival-Time Models† , 2014 .

[24]  H. Putter,et al.  Dynamic Prediction in Clinical Survival Analysis , 2011 .

[25]  Harald Binder,et al.  Quantifying the predictive accuracy of time‐to‐event models in the presence of competing risks , 2011, Biometrical journal. Biometrische Zeitschrift.

[26]  E Graf,et al.  Quantifying the Predictive Performance of Prognostic Models for Censored Survival Data with Time‐Dependent Covariates , 2008, Biometrics.

[27]  P. Heagerty,et al.  Survival Model Predictive Accuracy and ROC Curves , 2005, Biometrics.

[28]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[29]  M Jones,et al.  Accuracy of point predictions in survival analysis , 2001, Statistics in medicine.

[30]  Martha Sajatovic,et al.  Clinical Prediction Models , 2013 .

[31]  M. Schumacher,et al.  Consistent Estimation of the Expected Brier Score in General Survival Models with Right‐Censored Event Times , 2006, Biometrical journal. Biometrische Zeitschrift.

[32]  Leonhard Held,et al.  Objective Bayesian model selection for Cox regression , 2016, Statistics in medicine.

[33]  Elia Biganzoli,et al.  A time‐dependent discrimination index for survival data , 2005, Statistics in medicine.

[34]  John B. Willett,et al.  It’s About Time: Using Discrete-Time Survival Analysis to Study Duration and the Timing of Events , 1993 .

[35]  Martin Schumacher,et al.  Measures of prediction error for survival data with longitudinal covariates , 2011, Biometrical journal. Biometrische Zeitschrift.

[36]  P. Heagerty,et al.  Time‐Dependent Predictive Accuracy in the Presence of Competing Risks , 2010, Biometrics.

[37]  Matthias Schmid,et al.  Semiparametric regression for discrete time-to-event data , 2017, 1704.04087.

[38]  Tianxi Cai,et al.  Evaluating Prognostic Accuracy of Biomarkers under Competing Risk , 2012, Biometrics.