Assessing discriminative ability of risk models in clustered data

BackgroundThe discriminative ability of a risk model is often measured by Harrell’s concordance-index (c-index). The c-index estimates for two randomly chosen subjects the probability that the model predicts a higher risk for the subject with poorer outcome (concordance probability). When data are clustered, as in multicenter data, two types of concordance are distinguished: concordance in subjects from the same cluster (within-cluster concordance probability) and concordance in subjects from different clusters (between-cluster concordance probability). We argue that the within-cluster concordance probability is most relevant when a risk model supports decisions within clusters (e.g. who should be treated in a particular center). We aimed to explore different approaches to estimate the within-cluster concordance probability in clustered data.MethodsWe used data of the CRASH trial (2,081 patients clustered in 35 centers) to develop a risk model for mortality after traumatic brain injury. To assess the discriminative ability of the risk model within centers we first calculated cluster-specific c-indexes. We then pooled the cluster-specific c-indexes into a summary estimate with different meta-analytical techniques. We considered fixed effect meta-analysis with different weights (equal; inverse variance; number of subjects, events or pairs) and random effects meta-analysis. We reflected on pooling the estimates on the log-odds scale rather than the probability scale.ResultsThe cluster-specific c-index varied substantially across centers (IQR = 0.70-0.81; I2 = 0.76 with 95% confidence interval 0.66 to 0.82). Summary estimates resulting from fixed effect meta-analysis ranged from 0.75 (equal weights) to 0.84 (inverse variance weights). With random effects meta-analysis – accounting for the observed heterogeneity in c-indexes across clusters – we estimated a mean of 0.77, a between-cluster variance of 0.0072 and a 95% prediction interval of 0.60 to 0.95. The normality assumptions for derivation of a prediction interval were better met on the probability than on the log-odds scale.ConclusionWhen assessing the discriminative ability of risk models used to support decisions at cluster level we recommend meta-analysis of cluster-specific c-indexes. Particularly, random effects meta-analysis should be considered.

[1]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[2]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[3]  B. Jennett,et al.  Assessment of coma and impaired consciousness. A practical scale. , 1974, Lancet.

[4]  E. Lesaffre,et al.  An application of Harrell's C‐index to PH frailty models , 2010, Statistics in medicine.

[5]  Foss Mv.  MANAGEMENT OF VIRUS HEPATITIS. , 1964 .

[6]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[7]  Yvonne Vergouwe,et al.  Prediction models for clustered data: comparison of a random intercept and standard regression model , 2013, BMC Medical Research Methodology.

[8]  M. Gonen,et al.  Concordance probability and discriminatory power in proportional hazards regression , 2005 .

[9]  Yvonne Vergouwe,et al.  External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. , 2010, American journal of epidemiology.

[10]  Byung-Ho Nam,et al.  Discrimination Index, the Area Under the ROC Curve , 2002 .

[11]  B Jennett,et al.  Assessment of outcome after severe brain damage. , 1975, Lancet.

[12]  Paul Janssen,et al.  Frailty Model , 2007, International Encyclopedia of Statistical Science.

[13]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[14]  Raghu Kacker,et al.  Random-effects model for meta-analysis of clinical trials: an update. , 2007, Contemporary clinical trials.

[15]  Ewout Steyerberg,et al.  Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients , 2008, BMJ : British Medical Journal.

[16]  Dana Quade,et al.  Nonparametric Partial Correlation , 1967 .

[17]  Richard D Riley,et al.  Interpretation of random effects meta-analyses , 2011, BMJ : British Medical Journal.

[18]  C. Huber-Carol.  Goodness-of-Fit Tests and Model Validity , 2012 .

[19]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[20]  Juan Lu,et al.  Predicting Outcome after Traumatic Brain Injury: Development and International Validation of Prognostic Scores Based on Admission Characteristics , 2008, PLoS medicine.

[21]  S. Thompson,et al.  Detecting and describing heterogeneity in meta-analysis. , 1998, Statistics in medicine.

[22]  Peter Sandercock,et al.  Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months , 2005, The Lancet.

[23]  C.J.H. Mann,et al.  Clinical Prediction Models: A Practical Approach to Development, Validation and Updating , 2009 .

[24]  T. Alonzo.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating By Ewout W. Steyerberg , 2009 .

[25]  B. Jennett,et al.  ASSESSMENT OF OUTCOME AFTER SEVERE BRAIN DAMAGE A Practical Scale , 1975, The Lancet.

[26]  David J Spiegelhalter,et al.  A re-evaluation of random-effects meta-analysis , 2009, Journal of the Royal Statistical Society. Series A,.

[27]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[28]  M. Pencina,et al.  On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data , 2011, Statistics in medicine.

[29]  Emmanuel Lesaffre,et al.  Assessing the predictive ability of a multilevel binary regression model , 2012, Comput. Stat. Data Anal..

[30]  F. Harrell,et al.  Evaluating the yield of medical tests. , 1982, JAMA.

[31]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[32]  M. Pencina,et al.  Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation , 2004, Statistics in medicine.

[33]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.