Flexible recalibration of binary clinical prediction models

Calibration in binary prediction models, that is, the agreement between model predictions and observed outcomes, is an important aspect of assessing the models' utility for characterizing risk in future data. A popular technique for assessing model calibration first proposed by D. R. Cox in 1958 involves fitting a logistic model incorporating an intercept and a slope coefficient for the logit of the estimated probability of the outcome; good calibration is evident if these parameters do not appreciably differ from 0 and 1, respectively. However, in practice, the form of miscalibration may sometimes be more complicated. In this article, we expand the Cox calibration model to allow for more general parameterizations and derive a relative measure of miscalibration between two competing models from this more flexible model. We present an example implementation using data from the US Agency for Healthcare Research and Quality.

[1]  Armin Schubert,et al.  Broadly Applicable Risk Stratification System for Predicting Duration of Hospitalization and Mortality , 2010, Anesthesiology.

[2]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[3]  E. Mascha,et al.  Development and Validation of a Risk Quantification Index for 30-Day Postoperative Mortality and Morbidity in Noncardiac Surgical Patients , 2011, Anesthesiology.

[4]  Xiao-Hua Zhou,et al.  Statistical Methods in Diagnostic Medicine , 2002 .

[5]  A. Beckett,et al.  AKUFO AND IBARAPA. , 1965, Lancet.

[6]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[7]  G A Diamond,et al.  What price perfection? Calibration and discrimination of clinical prediction models. , 1992, Journal of clinical epidemiology.

[8]  M. Pepe,et al.  Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. , 2004, American journal of epidemiology.

[9]  J. C. van Houwelingen,et al.  A goodness-of-fit test for binary regression models, based on smoothing methods , 1991 .

[10]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[11]  F. Grover,et al.  Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. , 1997, Journal of the American College of Surgeons.

[12]  J. C. van Houwelingen,et al.  Predictive value of statistical models , 1990 .

[13]  M. Johantgen,et al.  Quality indicators using hospital discharge data: state and national applications. , 1998, The Joint Commission journal on quality improvement.

[14]  S. Lemeshow,et al.  European system for cardiac operative risk evaluation (EuroSCORE). , 1999, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[15]  W G Henderson,et al.  Assessment of predictive models for binary outcomes: an empirical approach using operative death from cardiac surgery. , 1994, Statistics in medicine.

[16]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[17]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[18]  M. Rosenthal,et al.  What works in market-oriented health policy? , 2009, The New England journal of medicine.

[19]  Michael W Kattan,et al.  Recent advances in evaluating the prognostic value of a marker , 2010, Scandinavian journal of clinical and laboratory investigation. Supplementum.

[20]  C J McDonald,et al.  Validation of Probabilistic Predictions , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[21]  D. Cox.  Two further applications of a model for binary regression , 1958 .