Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature

BackgroundWhile there is some consensus on methods for investigating statistical and methodological heterogeneity, little attention has been paid to clinical aspects of heterogeneity. The objective of this study is to summarize and collate suggested methods for investigating clinical heterogeneity in systematic reviews.MethodsWe searched databases (Medline, EMBASE, CINAHL, Cochrane Library, and CONSORT, to December 2010) and reference lists and contacted experts to identify resources providing suggestions for investigating clinical heterogeneity between controlled clinical trials included in systematic reviews. We extracted recommendations, assessed resources for risk of bias, and collated the recommendations.ResultsOne hundred and one resources were collected, including narrative reviews, methodological reviews, statistical methods papers, and textbooks. These resources generally had a low risk of bias, but there was minimal consensus among them. Resources suggested that planned investigations of clinical heterogeneity should be made explicit in the protocol of the review; clinical experts should be included on the review team; a set of clinical covariates should be chosen considering variables from the participant level, intervention level, outcome level, research setting, or others unique to the research question; covariates should have a clear scientific rationale; there should be a sufficient number of trials per covariate; and results of any such investigations should be interpreted with caution.ConclusionsThough the consensus was minimal, there were many recommendations in the literature for investigating clinical heterogeneity in systematic reviews. Formal recommendations for investigating clinical heterogeneity in systematic reviews of controlled trials are required.

[1]  C H Schmid,et al.  An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. , 1998, Statistics in medicine.

[2]  Adi Raveh,et al.  CoPlot: a tool for visualizing multivariate data in medicine. , 2008, Statistics in medicine.

[3]  K. Bailey,et al.  Inter-study differences: how should they influence the interpretation and analysis of results? , 1987, Statistics in medicine.

[4]  Raeburn Forbes,et al.  Systematic Reviews to Support Evidence-based Medicine , 2003, The Ulster Medical Journal.

[5]  D. Angus,et al.  Bench-to-bedside review: Avoiding pitfalls in critical care meta-analysis – funnel plots, risk estimates, types of heterogeneity, baseline risk and the ecologic fallacy , 2008, Critical care.

[6]  S J Pocock,et al.  Assessing treatment-time interaction in clinical trials with time to event data: a meta-analysis of hypertension trials. , 1998, Statistics in medicine.

[7]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[8]  William R. Shadish,et al.  Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues. , 1996 .

[9]  I Olkin,et al.  Diagnostic statistical procedures in medical meta-analyses. , 1999, Statistics in medicine.

[10]  Piero Quatto,et al.  Flexible meta-regression functions for modeling aggregate dose-response data, with an application to alcohol and mortality. , 2004, American journal of epidemiology.

[11]  Anne Whitehead,et al.  Meta-analysis of individual patient data versus aggregate data from longitudinal clinical trials , 2009, Clinical trials.

[12]  R. Rosenthal,et al.  Meta-analysis: recent developments in quantitative methods for literature reviews. , 2001, Annual review of psychology.

[13]  T Stijnen,et al.  Semi‐parametric modelling of the distribution of the baseline risk in meta‐analysis , 2007, Statistics in medicine.

[14]  Simon G Thompson,et al.  Can meta-analysis help target interventions at individuals most likely to benefit? , 2005, The Lancet.

[15]  Gordon Guyatt,et al.  Tips for learners of evidence-based medicine: 4. Assessing heterogeneity of primary studies in systematic reviews and whether to combine their results , 2005, Canadian Medical Association Journal.

[16]  Bertrand Baujat,et al.  A graphical method for exploring heterogeneity in meta‐analyses: application to a meta‐analysis of 65 trials , 2002, Statistics in medicine.

[17]  C. Naylor.  Two cheers for meta-analysis: problems and opportunities in aggregating results of clinical trials. , 1988, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[18]  G H Guyatt,et al.  A Consumer's Guide to Subgroup Analyses , 1992, Annals of Internal Medicine.

[19]  J. R. Scotti,et al.  Available From , 1973 .

[20]  Maroeska M Rovers,et al.  Subgroup effects despite homogeneous heterogeneity test results , 2010, BMC medical research methodology.

[21]  D. Sackett,et al.  Cochrane Collaboration , 1994, BMJ.

[22]  M. Cheung.  A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. , 2008, Psychological methods.

[23]  S G Thompson,et al.  Investigating underlying risk as a source of heterogeneity in meta-analysis. , 1997, Statistics in medicine.

[24]  Daniel S Nagin,et al.  Group-based trajectory modeling in clinical research. , 2010, Annual review of clinical psychology.

[25]  Evaluation of underlying risk as a source of heterogeneity in meta-analyses: a simulation study of Bayesian and frequentist implementations of three models. , 2007, Preventive veterinary medicine.

[26]  Nancy Santesso,et al.  Systematic reviews and knowledge translation. , 2006, Bulletin of the World Health Organization.

[27]  P. Shekelle,et al.  Principles of metaanalysis. , 2000, The Journal of rheumatology.

[28]  Paul Landais,et al.  Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. , 2004, Journal of clinical epidemiology.

[29]  Claire Glenton,et al.  Summaries of findings, descriptions of interventions, and information about adverse effects would make reviews more informative. , 2006, Journal of clinical epidemiology.

[30]  Andrea Furlan,et al.  Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group , 2003, Spine.

[31]  K Hemming,et al.  Meta-regression with partial information on summary trial or patient characteristics. , 2010, Statistics in medicine.

[32]  Saifudin Rashiq,et al.  Creating clinically relevant knowledge from systematic reviews: the challenges of knowledge translation. , 2007, Journal of evaluation in clinical practice.

[33]  I. König,et al.  Preferred reporting items of systematic review and meta-analyses: the PRISMA statement , 2011 .

[34]  Paula R Williamson,et al.  An overview of methods and empirical comparison of aggregate data and individual patient data results for investigating heterogeneity in meta-analysis of time-to-event outcomes. , 2005, Journal of Evaluation In Clinical Practice.

[35]  Akers,et al.  Systematic Reviews [ressource électronique]. CRD's guidance for undertaking reviews in health care , 2009 .

[36]  John P A Ioannidis,et al.  Interpretation of tests of heterogeneity and bias in meta-analysis. , 2008, Journal of evaluation in clinical practice.

[37]  L. Bouter,et al.  Clinical heterogeneity was a common problem in Cochrane reviews of physiotherapy and occupational therapy. , 2006, Journal of clinical epidemiology.

[38]  A. Laupacis,et al.  Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological Rigor , 2007, Annals of Internal Medicine.

[39]  J. Ioannidis,et al.  Predictive modeling and heterogeneity of baseline risk in meta-analysis of individual patient data. , 2001, Journal of clinical epidemiology.

[40]  R. Horwitz,et al.  Resolving conflicting clinical trials: guidelines for meta-analysis. , 1988, Journal of clinical epidemiology.

[41]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[42]  T Stijnen,et al.  Baseline risk as predictor of treatment benefit: three clinical meta-re-analyses. , 2000, Statistics in medicine.

[43]  T. Imperiale.  Meta-analysis: when and how. , 1999, Hepatology.

[44]  Kristian Thorlund,et al.  Attention should be given to multiplicity issues in systematic reviews. , 2008, Journal of clinical epidemiology.

[45]  Sunil J Rao,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2003 .

[46]  J P Pignon,et al.  Random effects survival models gave a better understanding of heterogeneity in individual patient data meta-analyses. , 2005, Journal of clinical epidemiology.

[47]  Julian P T Higgins,et al.  Controlling the risk of spurious findings from meta‐regression , 2004, Statistics in medicine.

[48]  P. Tugwell,et al.  Method guidelines for Cochrane Musculoskeletal Group systematic reviews. , 2006, The Journal of rheumatology.

[49]  J. Hilden,et al.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. , 2007, International journal of epidemiology.

[50]  S. Sharp,et al.  Explaining heterogeneity in meta-analysis: a comparison of methods. , 1997, Statistics in medicine.

[51]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[52]  D. Cook,et al.  Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. , 1995, Journal of clinical epidemiology.

[53]  D. Moher,et al.  Guides for reading and interpreting systematic reviews: III. How did the authors synthesize the data and make their conclusions? , 1998, Archives of pediatrics & adolescent medicine.

[54]  Jonathan A C Sterne,et al.  Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. , 2001, BMJ.

[55]  J. Ioannidis,et al.  Quantitative Synthesis in Systematic Reviews , 1997, Annals of Internal Medicine.

[56]  A Whitehead,et al.  Meta‐analysis of continuous outcome data from individual patients , 2001, Statistics in medicine.

[57]  A. Brennan,et al.  PAR15 USING MIXED TREATMENT COMPARISONS AND METAREGRESSION TO PERFORM INDIRECT COMPARISONS TO ESTIMATE THE EFFICACY OF BIOLOGIC TREATMENTS IN RHEUMATOID ARTHRITIS , 2007 .

[58]  P. Glasziou,et al.  Investigating causes of heterogeneity in systematic reviews , 2002, Statistics in medicine.

[59]  George Davey Smith,et al.  Meta-analysis: Beyond the grand mean? , 1997, BMJ.

[60]  Paula R Williamson,et al.  Investigating heterogeneity in an individual patient data meta‐analysis of time to event outcomes , 2005, Statistics in medicine.

[61]  G. Colditz,et al.  How to review the evidence: systematic identification and review of the scientific literature , 2000 .

[62]  William D Fraser,et al.  Exploring heterogeneity in meta-analyses: needs, resources and challenges. , 2008, Paediatric and perinatal epidemiology.

[63]  C D Naylor,et al.  Meta-analysis of controlled clinical trials. , 1989, The Journal of rheumatology.

[64]  P Peduzzi,et al.  Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy. , 1995, Journal of clinical epidemiology.

[65]  P C Lambert,et al.  A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. , 2002, Journal of clinical epidemiology.

[66]  B H Chang,et al.  Meta‐analysis of binary data: which within study variance estimate to use? , 2001, Statistics in medicine.

[67]  D. Toobert,et al.  Mediation analyses: applications in nutrition research and reading the literature. , 2010, Journal of the American Dietetic Association.

[68]  M C Simmonds,et al.  Covariate heterogeneity in meta‐analysis: Criteria for deciding between meta‐regression and individual patient data , 2007, Statistics in medicine.

[69]  Andrew Herxheimer,et al.  Systematic reviews of adverse effects: framework for a structured approach , 2007, BMC medical research methodology.

[70]  Meera Viswanathan,et al.  Comparative Effectiveness Review Methods: Clinical Heterogeneity , 2010 .

[71]  M Bateson,et al.  Systematic Reviews to Support Evidence-Based Medicine: How to Review and Apply Findings of Healthcare Research. , 2004 .

[72]  G G Koch,et al.  An overview of statistical issues and methods of meta-analysis. , 1991, Journal of biopharmaceutical statistics.

[73]  R J Cook,et al.  A logistic model for trend in 2 x 2 x kappa tables with applications to meta-analyses. , 1997, Biometrics.

[74]  Julian P T Higgins,et al.  A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. , 2009, Journal of clinical epidemiology.

[75]  David R. Jones,et al.  Systematic reviews of trials and other studies. , 1998, Health technology assessment.

[76]  M. McIntosh,et al.  The population risk as an explanatory variable in research synthesis of clinical trials. , 1996, Statistics in medicine.

[77]  Harold I Feldman,et al.  Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head , 2002, Statistics in medicine.

[78]  J. Karlberg,et al.  Assessing heterogeneity in meta-analyses of Helicobacter pylori infection-related clinical studies: a critical appraisal. , 2004, Chinese journal of digestive diseases.

[79]  S. Thompson,et al.  How should meta‐regression analyses be undertaken and interpreted? , 2002, Statistics in medicine.

[80]  Judith A. Hall,et al.  Interpreting and Evaluating Meta-Analysis , 1995, Evaluation & the health professions.

[81]  S D Walter,et al.  Variation in baseline risk as an explanation of heterogeneity in meta-analysis. , 1997, Statistics in medicine.

[82]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[83]  F Song,et al.  Exploring heterogeneity in meta-analysis: is the L'Abbé plot useful? , 1999, Journal of clinical epidemiology.

[84]  D. Altman,et al.  Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice , 2002, Journal of health services research & policy.

[85]  J. Berlin,et al.  Invited commentary: benefits of heterogeneity in meta-analysis of data from epidemiologic studies. , 1995, American journal of epidemiology.

[86]  Carl Heneghan,et al.  What is missing from descriptions of treatment in trials and reviews? , 2008, BMJ : British Medical Journal.

[87]  Sophie Hill,et al.  Taking healthcare interventions from trial to practice , 2010, BMJ : British Medical Journal.

[88]  David Moher,et al.  An empirical study using permutation-based resampling in meta-regression , 2012, Systematic Reviews.

[89]  R. Bender,et al.  Intervention Effects in the Case of Heterogeneity between Three Subgroups , 2010, Methods of Information in Medicine.

[90]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[91]  J. Cappelleri,et al.  Understanding heterogeneity in meta‐analysis: the role of meta‐regression , 2009, International journal of clinical practice.

[92]  J. Concato,et al.  Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. , 1995, Journal of clinical epidemiology.

[93]  A Whitehead,et al.  Borrowing strength from external trials in a meta-analysis. , 1996, Statistics in medicine.

[94]  Mark C Simmonds,et al.  Meta-analysis of individual patient data from randomized trials: a review of methods used in practice , 2005, Clinical trials.

[95]  L. Moja,et al.  Heterogeneity and meta-analyses: do study results truly differ? , 2009, Internal and emergency medicine.

[96]  Richard D Riley,et al.  Meta‐analysis of continuous outcomes combining individual patient data and aggregate data , 2008, Statistics in medicine.

[97]  C S Berkey,et al.  Multiple-outcome meta-analysis of clinical trials. , 1996, Statistics in medicine.

[98]  Alex J. Sutton,et al.  Methods for Meta-Analysis in Medical Research , 2000 .

[99]  S G Thompson,et al.  Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches. , 2000, Statistics in medicine.

[100]  C. Frost,et al.  Use of hierarchical models for meta-analysis: experience in the metabolic ward studies of diet and blood cholesterol. , 1999, Statistics in medicine.

[101]  Douglas G Altman,et al.  Bayesian random effects meta‐analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales by D. E. Warn, S. G. Thompson and D. J. Spiegelhalter, Statistics in Medicine 2002; 21: 1601–1623 , 2005, Statistics in medicine.

[102]  Nancy Santesso,et al.  Knowledge for knowledge translation: The role of the Cochrane Collaboration , 2006, The Journal of continuing education in the health professions.

[103]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[104]  C S Berkey,et al.  A random-effects regression model for meta-analysis. , 1995, Statistics in medicine.

[105]  J. Concato,et al.  A simulation study of the number of events per variable in logistic regression analysis. , 1996, Journal of clinical epidemiology.

[106]  R M Turner,et al.  Multilevel models for meta-analysis, and their application to absolute risk differences , 2001, Statistical methods in medical research.

[107]  David Moher,et al.  An international registry of systematic-review protocols , 2011, The Lancet.

[108]  N. Freemantle,et al.  DERIVING TREATMENT RECOMMENDATIONS FROM EVIDENCE WITHIN RANDOMIZED TRIALS The Role and Limitation of Meta-Analysis , 1999, International Journal of Technology Assessment in Health Care.

[109]  Julian P T Higgins,et al.  Recent developments in meta‐analysis , 2008, Statistics in medicine.

[110]  Theo Stijnen,et al.  Advanced methods in meta‐analysis: multivariate approach and meta‐regression , 2002, Statistics in medicine.

[111]  Kurex Sidik,et al.  A Note on Variance Estimation in Random Effects Meta-Regression , 2005, Journal of biopharmaceutical statistics.

[112]  P. Glasziou,et al.  Avoidable waste in the production and reporting of research evidence , 2009, The Lancet.

[113]  A. B. Thomsen,et al.  Heterogeneity can impair the results of Cochrane meta‐analyses despite accordance with statistical guidelines , 2008, Allergy.

[114]  S G Thompson,et al.  Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated , 1994, BMJ.

[115]  David R. Jones,et al.  Bayesian methods in health technology assessment: a review. , 2000, Health technology assessment.

[116]  K R Abrams,et al.  Methods for exploring heterogeneity in meta-analysis. , 2001, Evaluation & the health professions.

[117]  Diederick E Grobbee,et al.  A systematic review of analytical methods used to study subgroups in (individual patient data) meta-analyses. , 2007, Journal of clinical epidemiology.

[118]  Claire Bombardier,et al.  2009 Updated Method Guidelines for Systematic Reviews in the Cochrane Back Review Group , 2009, Spine.

[119]  C Anello,et al.  Exploratory or analytic meta-analysis: should we distinguish between them? , 1995, Journal of clinical epidemiology.

[120]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration , 2013 .

[121]  N R St-Pierre,et al.  Invited review: Integrating quantitative findings from multiple studies using mixed model methodology. , 2001, Journal of dairy science.

[122]  M. Lipsey,et al.  The Way in Which Intervention Studies Have “Personality” and why it is Important to Meta-Analysis , 2001, Evaluation & the health professions.