Individual participant data meta-analyses compared with meta-analyses based on aggregate data

BACKGROUND Meta-analyses based on individual participant data (IPD-MAs) allow more powerful and uniformly consistent analyses as well as better characterisation of subgroups and outcomes, compared to those which are based on aggregate data (AD-MAs) extracted from published trial reports. However, IPD-MAs are a larger undertaking requiring greater resources than AD-MAs. Researchers have compared results from IPD-MA against results obtained from AD-MA and reported conflicting findings. We present a methodology review to summarise this empirical evidence . OBJECTIVES To review systematically empirical comparisons of meta-analyses of randomised trials based on IPD with those based on AD extracted from published reports, to evaluate the level of agreement between IPD-MA and AD-MA and whether agreement is affected by differences in type of effect measure, trials and participants included within the IPD-MA and AD-MA, and whether analyses were undertaken to explore the main effect of treatment or a treatment effect modifier. SEARCH METHODS An electronic search of the Cochrane Library (includes Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effectiveness, CENTRAL, Cochrane Methodology Register, HTA database, NHS Economic Evaluations Database), MEDLINE, and Embase was undertaken up to 7 January 2016. Potentially relevant articles that were known to any of the review authors and reference lists of retrieved articles were also checked. SELECTION CRITERIA Studies reporting an empirical comparison of the results of meta-analyses of randomised trials using IPD with those using AD. Studies were included if sufficient numerical data, comparing IPD-MA and AD-MA, were available in their reports. DATA COLLECTION AND ANALYSIS Two review authors screened the title and abstract of identified studies with full-text publications retrieved for those identified as eligible or potentially eligible. A 'quality' assessment was done and data were extracted independently by two review authors with disagreements resolved by involving a third author. Data were summarised descriptively for comparisons where an estimate of effect measure and corresponding precision have been provided both for IPD-MA and for AD-MA in the study report. Comparisons have been classified according to whether identical effect measures, identical trials and patients had been used in the IPD-MA and the AD-MA, and whether the analyses were undertaken to explore the main effect of treatment, or to explore a potential treatment effect modifier.Effect measures were transformed to a standardised scale (z scores) and scatter plots generated to allow visual comparisons. For each comparison, we compared the statistical significance (at the 5% two-sided level) of an IPD-MA compared to the corresponding AD-MA and calculated the number of discrepancies. We examined discrepancies by type of analysis (main effect or modifier) and according to whether identical trials, patients and effect measures had been used by the IPD-MA and AD-MA. We calculated the average of differences between IPD-MA and AD-MA (z scores, ratio effect estimates and standard errors (of ratio effects)) and 95% limits of agreement. MAIN RESULTS From the 9330 reports found by our searches, 39 studies were eligible for this review with effect estimate and measure of precision extracted for 190 comparisons of IPD-MA and AD-MA. We classified the quality of studies as 'no important flaws' (29 (74%) studies) or 'possibly important flaws' (10 (26%) studies).A median of 4 (interquartile range (IQR): 2 to 6) comparisons were made per study, with 6 (IQR 4 to 11) trials and 1225 (542 to 2641) participants in IPD-MAs and 7 (4 to 11) and 1225 (705 to 2541) for the AD-MAs. One hundred and forty-four (76%) comparisons were made on the main treatment effect meta-analysis and 46 (24%) made using results from analyses to explore treatment effect modifiers.There is agreement in statistical significance between the IPD-MA and AD-MA for 152 (80%) comparisons, 23 of which disagreed in direction of effect. There is disagreement in statistical significance for 38 (20%) comparisons with an excess proportion of IPD-MA detecting a statistically significant result that was not confirmed with AD-MA (28 (15%)), compared with 10 (5%) comparisons with a statistically significant AD-MA that was not confirmed by IPD-MA. This pattern of disagreement is consistent for the 144 main effect analyses but not for the 46 comparisons of treatment effect modifier analyses. Conclusions from some IPD-MA and AD-MA differed even when based on identical trials, participants (but not necessarily identical follow-up) and treatment effect measures. The average difference between IPD-MA and AD-MA in z scores, ratio effect estimates and standard errors is small but limits of agreement are wide and include important differences in both directions. Discrepancies between IPD-MA and AD-MA do not appear to increase as the differences between trials and participants increase. AUTHORS' CONCLUSIONS IPD offers the potential to explore additional, more thorough, and potentially more appropriate analyses compared to those possible with AD. But in many cases, similar results and conclusions can be drawn from IPD-MA and AD-MA. Therefore, before embarking on a resource-intensive IPD-MA, an AD-MA should initially be explored and researchers should carefully consider the potential added benefits of IPD.

[1]  M. Clarke,et al.  Systematic reviews using individual patient data: a map for the minefields? , 1998, Annals of oncology : official journal of the European Society for Medical Oncology.

[2]  M. Sampson,et al.  Checking reference lists to find additional studies for systematic reviews. , 2011, The Cochrane database of systematic reviews.

[3]  L. Stewart,et al.  Feasibility and reliability of using hazard ratios in meta-analyses of published time-to-event data , 2001 .

[4]  C. Davis,et al.  Palliative chemotherapy for advanced or metastatic colorectal cancer. Colorectal Meta-analysis Collaboration. , 2000, The Cochrane database of systematic reviews.

[5]  P. Rothwell,et al.  Effect of daily aspirin on long-term risk of death due to cancer: analysis of individual patient data from randomised trials , 2011, The Lancet.

[6]  R M Turner,et al.  Multilevel models for meta-analysis, and their application to absolute risk differences , 2001, Statistical methods in medical research.

[7]  L F Burmeister,et al.  A comparison of meta-analytic results using literature vs individual patient data. Paternal cell immunization for recurrent miscarriage. , 1995, JAMA.

[8]  Charles K. Cooper,et al.  Meta-analysis of a possible signal of increased mortality associated with cefepime use. , 2010, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[9]  I Olkin,et al.  Comparison of effect estimates from a meta-analysis of summary data from published studies and from a meta-analysis using individual patient data for ovarian cancer studies. , 1997, American journal of epidemiology.

[10]  Junichi Sakamoto,et al.  Individual patient-level and study-level meta-analysis for investigating modifiers of treatment effect. , 2004, Japanese journal of clinical oncology.

[11]  Stefan Michiels,et al.  Meta-analysis when only the median survival times are known: A comparison with individual patient data results , 2005, International Journal of Technology Assessment in Health Care.

[12]  J. Bohlius,et al.  Ten years of meta-analyses on erythropoiesis-stimulating agents in cancer patients. , 2011, Cancer treatment and research.

[13]  L. Santoro,et al.  P118 Prospective meta-analysis using individual patient data vs meta-analysis of published reports: The case of ace-inhibitors in myocardial infarction , 1997 .

[14]  Harold I Feldman,et al.  Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head , 2002, Statistics in medicine.

[15]  Paul Landais,et al.  Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. , 2004, Journal of clinical epidemiology.

[16]  A. Sutton,et al.  Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey , 2012, BMJ : British Medical Journal.

[17]  A. Hoes,et al.  Empirical comparison of subgroup effects in conventional and individual patient data meta-analyses , 2008, International Journal of Technology Assessment in Health Care.

[18]  R. Salamon,et al.  [Effect of dapsone on survival in HIV infected patients: a meta- analysis of finished trials]. , 2000, Revue d'epidemiologie et de sante publique.

[19]  Mike Clarke,et al.  Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: an overview of 6,633 patients from 27 randomized trials. Myeloma Trialists' Collaborative Group. , 1998, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[20]  L. Beckett,et al.  Validation of a meta-analysis: the effects of fish oil in rheumatoid arthritis. , 1995, Journal of clinical epidemiology.

[21]  T. Mathew,et al.  On the Equivalence of Meta‐Analysis Using Literature and Using Individual Patient Data , 1999, Biometrics.

[22]  J. Crawford,et al.  Benefits and risks of using erythropoiesis-stimulating agents (ESAs) in lung cancer patients: study-level and patient-level meta-analyses. , 2012, Lung cancer.

[23]  J. Manson,et al.  Vitamin D with calcium reduces mortality: patient level pooled analysis of 70,528 patients from eight major vitamin D trials. , 2012, The Journal of clinical endocrinology and metabolism.

[24]  J. Gladman,et al.  Hospital at home early discharge , 2009 .

[25]  Anne Whitehead,et al.  Meta-analysis of individual patient data versus aggregate data from longitudinal clinical trials , 2009, Clinical trials.

[26]  Neil Marlow,et al.  Elective high-frequency oscillatory versus conventional ventilation in preterm infants: a systematic review and meta-analysis of individual patients' data , 2010, The Lancet.

[27]  J. Camm,et al.  Effect of fish oil on ventricular tachyarrhythmia in three studies in patients with implantable cardioverter defibrillators. , 2008, European heart journal.

[28]  Paula R Williamson,et al.  An overview of methods and empirical comparison of aggregate data and individual patient data results for investigating heterogeneity in meta-analysis of time-to-event outcomes. , 2005, Journal of Evaluation In Clinical Practice.

[29]  V Torri,et al.  Effectiveness of antibiotic prophylaxis in critically ill adult patients: systematic review of randomised controlled trials , 1998, BMJ.

[30]  L. Stewart,et al.  Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. , 1995, Statistics in medicine.

[31]  I. Chalmers.  The Cochrane Collaboration: Preparing, Maintaining, and Disseminating Systematic Reviews of the Effects of Health Care , 1993, Annals of the New York Academy of Sciences.

[32]  H Goldstein,et al.  A multilevel model framework for meta-analysis of clinical trials with binary outcomes. , 2000, Statistics in medicine.

[33]  I Olkin,et al.  Comparison of meta-analysis versus analysis of variance of individual patient data. , 1998, Biometrics.

[34]  M. Parmar,et al.  Meta-analysis of the literature or of individual patient data: is there a difference? , 1993, The Lancet.

[35]  J L Hutton,et al.  Individual patient data meta-analysis of randomized anti-epileptic drug monotherapy trials. , 2000, Journal of evaluation in clinical practice.

[36]  J. Ioannidis,et al.  Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. , 1999, Journal of clinical epidemiology.

[37]  Paula R. Williamson,et al.  The value of the aggregate data approach in meta‐analysis with time‐to‐event outcomes , 2001 .

[38]  J. Berlin,et al.  The Effect of Antilymphocyte Induction Therapy on Renal Allograft Survival , 1998, Annals of Internal Medicine.

[39]  J. Jansen,et al.  Efficacy of once-daily indacaterol 75 μg relative to alternative bronchodilators in COPD: A study level and a patient level network meta-analysis , 2012, BMC Pulmonary Medicine.

[40]  Catrin Tudur Smith,et al.  Combining individual patient data and aggregate data in mixed treatment comparison meta‐analysis: Individual patient data may be beneficial if only for a subset of trials , 2013, Statistics in medicine.

[41]  J. Pignon,et al.  Individual patient-versus literature-based meta-analysis of survival data: time to event and event rate at a particular time can make a difference, an example based on head and neck cancer. , 2001, Controlled clinical trials.

[42]  P C Lambert,et al.  A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. , 2002, Journal of clinical epidemiology.

[43]  J. Trachtenberg,et al.  Maximal androgen blockade for the treatment of metastatic prostate cancer—a systematic review , 2006, Current oncology.

[44]  Harris Cooper,et al.  The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. , 2009, Psychological methods.

[45]  P. Sandercock,et al.  Alteplase and ischaemic stroke: have new reviews of old data helped? , 2005, The Lancet Neurology.

[46]  R. Arriagada,et al.  Role of thoracic radiotherapy in limited-stage small-cell lung cancer: quantitative review based on the literature versus meta-analysis based on individual data. , 1992, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[47]  T. Chevalier.  Chemotherapy for Advanced NSCLC , 1996 .