Multiple Score Comparison: a network meta-analysis approach to comparison and external validation of prognostic scores

BackgroundPrediction models and prognostic scores have been increasingly popular in both clinical practice and clinical research settings, for example to aid in risk-based decision making or control for confounding. In many medical fields, a large number of prognostic scores are available, but practitioners may find it difficult to choose between them due to lack of external validation as well as lack of comparisons between them.MethodsBorrowing methodology from network meta-analysis, we describe an approach to Multiple Score Comparison meta-analysis (MSC) which permits concurrent external validation and comparisons of prognostic scores using individual patient data (IPD) arising from a large-scale international collaboration. We describe the challenges in adapting network meta-analysis to the MSC setting, for instance the need to explicitly include correlations between the scores on a cohort level, and how to deal with many multi-score studies. We propose first using IPD to make cohort-level aggregate discrimination or calibration scores, comparing all to a common comparator. Then, standard network meta-analysis techniques can be applied, taking care to consider correlation structures in cohorts with multiple scores. Transitivity, consistency and heterogeneity are also examined.ResultsWe provide a clinical application, comparing prognostic scores for 3-year mortality in patients with chronic obstructive pulmonary disease using data from a large-scale collaborative initiative. We focus on the discriminative properties of the prognostic scores. Our results show clear differences in performance, with ADO and eBODE showing higher discrimination with respect to mortality than other considered scores. The assumptions of transitivity and local and global consistency were not violated. Heterogeneity was small.ConclusionsWe applied a network meta-analytic methodology to externally validate and concurrently compare the prognostic properties of clinical scores. Our large-scale external validation indicates that the scores with the best discriminative properties to predict 3 year mortality in patients with COPD are ADO and eBODE.

[1]  M. Martínez-García,et al.  Severe exacerbations and BODE index: two independent risk factors for death in male COPD patients. , 2009, Respiratory medicine.

[2]  H. Roslan,et al.  The SAFE (SGRQ score, air-flow limitation and exercise tolerance) Index: a new composite score for the stratification of severity in chronic obstructive pulmonary disease , 2007, Postgraduate Medical Journal.

[3]  G. Collins,et al.  External validation of multivariable prediction models: a systematic review of methodological conduct and reporting , 2014, BMC Medical Research Methodology.

[4]  S Dias,et al.  Checking consistency in mixed treatment comparison meta‐analysis , 2010, Statistics in medicine.

[5]  N. Welton,et al.  Addressing between‐study heterogeneity and inconsistency in mixed treatment comparisons: Application to stroke prevention treatments in individuals with non‐rheumatic atrial fibrillation , 2009, Statistics in medicine.

[6]  François Gueyffier,et al.  Uptake of systematic reviews and meta-analyses based on individual participant data in clinical practice guidelines: descriptive study , 2015, BMJ : British Medical Journal.

[7]  G. Riet,et al.  COPD: more than respiratory , 2008, The Lancet.

[8]  Andrea Cipriani,et al.  Conceptual and Technical Challenges in Network Meta-analysis , 2013, Annals of Internal Medicine.

[9]  Jesse A. Berlin,et al.  Assessing the Generalizability of Prognostic Information , 1999 .

[10]  T. Johnson,et al.  Group art therapy as an adjunctive treatment for people with schizophrenia: multicentre pragmatic randomised trial , 2012, BMJ : British Medical Journal.

[11]  Ulrike Held,et al.  Expansion of the prognostic assessment of patients with chronic obstructive pulmonary disease: the updated BODE index and the ADO index , 2009, The Lancet.

[12]  J. Wyatt,et al.  Commentary: Prognostic models: clinically useful or quickly forgotten? , 1995 .

[13]  D. Mark,et al.  Clinical prediction models: are we building better mousetraps? , 2003, Journal of the American College of Cardiology.

[14]  Margaret S Pepe,et al.  Problems with risk reclassification methods for evaluating prediction models. , 2011, American journal of epidemiology.

[15]  Camilla Bianchi,et al.  Prediction models for exacerbations in patients with COPD , 2017, European Respiratory Review.

[16]  J. Soriano,et al.  Multicomponent indices to predict survival in COPD: the COCOMICS study , 2012, European Respiratory Journal.

[17]  Lucas M Bachmann,et al.  Individual patient data meta-analysis of diagnostic and prognostic studies in obstetrics, gynaecology and reproductive medicine , 2009, BMC medical research methodology.

[18]  Catrin Tudur Smith,et al.  Assessing key assumptions of network meta‐analysis: a review of methods , 2013, Research synthesis methods.

[19]  A Rogier T Donders,et al.  Dealing with missing outcome data in randomized trials and observational studies. , 2012, American journal of epidemiology.

[20]  Dimitris Mavridis,et al.  Demystifying fixed and random effects meta-analysis , 2014, Evidence-Based Mental Health.

[21]  O. Panagiotou.  Network Meta-Analysis: Evidence Synthesis With Mixed Treatment Comparison Edited by Giuseppe Biondi-Zoccai , 2015 .

[22]  Huseyin Naci,et al.  Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers , 2013, BMC Medicine.

[23]  M. Puhan,et al.  Prediction of COPD-specific health-related quality of life in primary care COPD patients: a prospective cohort study , 2014, npj Primary Care Respiratory Medicine.

[24]  Gary S Collins,et al.  Comparing risk prediction models , 2012, BMJ : British Medical Journal.

[25]  Richard D Riley,et al.  Developing and validating risk prediction models in an individual participant data meta-analysis , 2014, BMC Medical Research Methodology.

[26]  Shannon Cope,et al.  Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes , 2012, BMC Medical Research Methodology.

[27]  E. Steyerberg,et al.  Clinical usefulness of the Framingham cardiovascular risk profile beyond its statistical performance: the Tehran Lipid and Glucose Study. , 2012, American journal of epidemiology.

[28]  Ciro Casanova,et al.  The body-mass index, airflow obstruction, dyspnea, and exercise capacity index in chronic obstructive pulmonary disease. , 2004, The New England journal of medicine.

[29]  I Olkin,et al.  Simple Pooling versus Combining in Meta-Analysis , 2001, Evaluation & the health professions.

[30]  Yemisi Takwoingi,et al.  Empirical Evidence of the Importance of Comparative Studies of Diagnostic Test Accuracy , 2013, Annals of Internal Medicine.

[31]  Johannes B Reitsma,et al.  GetReal in network meta‐analysis: a review of the methodology , 2016, Research synthesis methods.

[32]  Niels H Chavannes,et al.  Derivation and validation of a composite index of severity in chronic obstructive pulmonary disease: the DOSE Index. , 2009, American journal of respiratory and critical care medicine.

[33]  S. Thompson,et al.  How should meta‐regression analyses be undertaken and interpreted? , 2002, Statistics in medicine.

[34]  S. MacMahon,et al.  The ADVANCE trial – Authors' reply , 2008, The Lancet.

[35]  Dimitris Mavridis,et al.  How to interpret meta-analysis models: fixed effect and random effects meta-analyses , 2014, Evidence-Based Mental Health.

[36]  M. Bennett,et al.  Cancer induced bone pain , 2015, BMJ : British Medical Journal.

[37]  T. Lumley.  Network meta‐analysis for indirect treatment comparisons , 2002, Statistics in medicine.

[38]  Tim Peakman,et al.  New Models for Large Prospective Studies: Is There a Better Way? , 2012, American journal of epidemiology.

[39]  Yvonne Vergouwe,et al.  External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. , 2010, American journal of epidemiology.

[40]  F. Dominici,et al.  Reproducible epidemiologic research. , 2006, American journal of epidemiology.

[41]  Stephen R Cole,et al.  Accounting for misclassified outcomes in binary regression models using multiple imputation with internal validation data. , 2013, American journal of epidemiology.

[42]  Alice M Turner,et al.  Mortality prediction in chronic obstructive pulmonary disease comparing the GOLD 2007 and 2011 staging systems: a pooled analysis of individual patient data. , 2015, The Lancet. Respiratory medicine.

[43]  Nicky J Welton,et al.  Linear inference for mixed treatment comparison meta‐analysis: A two‐stage approach , 2011, Research synthesis methods.

[44]  G. Lu,et al.  Combination of direct and indirect evidence in mixed treatment comparisons , 2004, Statistics in medicine.

[45]  Sharon E. Straus,et al.  Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review , 2017, BMC Medicine.

[46]  Kristian Thorlund,et al.  The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations , 2015, Annals of Internal Medicine.

[47]  A E Ades,et al.  Accounting for correlation in network meta‐analysis with multi‐arm trials , 2012, Research synthesis methods.

[48]  J A Cook,et al.  The rise of big clinical databases , 2015, The British journal of surgery.

[49]  J. Hanley,et al.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases. , 1983, Radiology.

[50]  T. Alonzo.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating By Ewout W. Steyerberg , 2009 .

[51]  Richard D Riley,et al.  Performance of methods for meta-analysis of diagnostic test accuracy with few studies or sparse data , 2015, Statistical methods in medical research.

[52]  Douglas G Altman,et al.  Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study , 2010, BMC medical research methodology.

[53]  Ian R. White,et al.  Network Meta-analysis , 2015 .

[54]  Georgia Salanti,et al.  Evaluation of networks of randomized trials , 2008, Statistical methods in medical research.

[55]  J. Soriano,et al.  Prognostic assessment in COPD without lung function: the B-AE-D indices , 2016, European Respiratory Journal.

[56]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[57]  G. Collins,et al.  Prediction models for cardiovascular disease risk in the general population: systematic review , 2016, British Medical Journal.

[58]  Karel G M Moons,et al.  A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta‐analysis , 2013, Statistics in medicine.

[59]  A. Sutton,et al.  Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey , 2012, BMJ : British Medical Journal.

[60]  A Whitehead,et al.  Borrowing strength from external trials in a meta-analysis. , 1996, Statistics in medicine.

[61]  Mark Woodward,et al.  Assessing Risk Prediction Models Using Individual Participant Data From Multiple Studies , 2013, American journal of epidemiology.

[62]  G. Lu,et al.  Assessing Evidence Inconsistency in Mixed Treatment Comparisons , 2006 .

[63]  N. Marlow,et al.  Systematic Reviews and Meta-and Pooled Analyses Risk Factor Models for Neurodevelopmental Outcomes in Children Born Very Preterm or With Very Low Birth Weight : A Systematic Review of Methodology and Reporting , 2017 .

[64]  F. Martinez,et al.  Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: GOLD executive summary. , 2007, American journal of respiratory and critical care medicine.

[65]  Georgia Salanti,et al.  Indirect and mixed‐treatment comparison, network, or multiple‐treatments meta‐analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool , 2012, Research synthesis methods.

[66]  Harlan M Krumholz,et al.  Why data sharing should be the expected norm , 2015, BMJ : British Medical Journal.

[67]  Johannes B. Reitsma,et al.  Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use , 2015, PLoS medicine.

[68]  Anna Chaimani,et al.  Evaluating the Quality of Evidence from a Network Meta-Analysis , 2014, PloS one.

[69]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[70]  Margaret McCartney,et al.  Margaret McCartney: Jeremy Hunt should follow his own advice on emergency care , 2015, BMJ : British Medical Journal.

[71]  Wei Bao,et al.  Systematic Reviews and Meta-and Pooled Analyses Predicting Risk of Type 2 Diabetes Mellitus with Genetic Risk Models on the Basis of Established Genome-wide Association Markers : A Systematic Review , 2013 .

[72]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[73]  M Blettner,et al.  Traditional reviews, meta-analyses and pooled analyses in epidemiology. , 1999, International journal of epidemiology.