Heterogeneity: Subgroups, Meta-Regression, Bias And Bias-Adjustment

The NICE Guide to the Methods of Technology Appraisal i is a regularly updated document that provides an overview of the key principles and methods of health technology assessment and appraisal for use in NICE appraisals. The Methods Guide does not provide detailed advice on how to implement and apply the methods it describes. This DSU series of Technical Support Documents (TSDs) is intended to complement the Methods Guide by providing detailed information on how to implement specific methods. The TSDs provide a review of the current state of the art in each topic area, and make clear recommendations on the implementation of methods and reporting standards where it is appropriate to do so. They aim to provide assistance to all those involved in submitting or critiquing evidence as part of NICE Technology Appraisals, whether manufacturers, assessment groups or any other stakeholder type. We recognise that there are areas of uncertainty, controversy and rapid development. It is our intention that such areas are indicated in the TSDs. All TSDs are extensively peer reviewed prior to publication (the names of peer reviewers appear in the acknowledgements for each document). Nevertheless, the responsibility for each TSD lies with the authors and we welcome any constructive feedback on the content or suggestions for further guides. Please be aware that whilst the DSU is funded by NICE, these documents do not constitute formal NICE guidance or policy. Excellence (NICE) through its Decision Support Unit. The views, and any errors or omissions, expressed in this document are of the authors only. NICE may take account of part or all of this document if it considers it appropriate, but it is not bound to do so. This Technical Support Document focuses on heterogeneity in relative treatment effects. Heterogeneity indicates the presence of effect-modifiers. A distinction is usually made between true variability in treatment effects due to variation between patient populations or settings, and biases related to the way in which trials were conducted. Variability in relative treatment effects threatens the external validity of trial evidence, and limits the ability to generalise from the results, imperfections in trial conduct represent threats to internal validity. In either case it is emphasised that, although we continue to focus attention on evidence from trials, the study of effect-modifying covariates is in every way a form of observational study, because patients cannot be randomised to covariate values. This document provides guidance …

[1]  A. Sutton,et al.  Mixed treatment comparisons using aggregate and individual participant level data , 2012, Statistics in medicine.

[2]  Nicky J Welton,et al.  Evaluating novel agent effects in multiple‐treatments meta‐regression , 2010, Statistics in medicine.

[3]  Adrian F. M. Smith,et al.  Bayesian Statistics 5. , 1998 .

[4]  Simon G Thompson,et al.  Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews , 2012, International journal of epidemiology.

[5]  F Dominici,et al.  Combining Contingency Tables with Missing Dimensions , 2000, Biometrics.

[6]  H Goldstein,et al.  A multilevel model framework for meta-analysis of clinical trials with binary outcomes. , 2000, Statistics in medicine.

[7]  A. Sutton,et al.  Meta-analysis methods for combining information from different sources in evaluating health interventions , 2002 .

[8]  P C Lambert,et al.  A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. , 2002, Journal of clinical epidemiology.

[9]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[10]  Ross J. Harris,et al.  Development of a combined database for meta‐epidemiological research , 2010, Research synthesis methods.

[11]  Mark C Simmonds,et al.  Meta-analysis of individual patient data from randomized trials: a review of methods used in practice , 2005, Clinical trials.

[12]  Catrin Tudur-Smith Combining individual patient data and aggregate data in mixed treatment comparison meta-analysis: individual patient data may be beneficial if only for a subset of trials. , 2012 .

[13]  José M. Bernardo,et al.  Bayesian Statistics , 2011, International Encyclopedia of Statistical Science.

[14]  Paula R Williamson,et al.  Investigating heterogeneity in an individual patient data meta‐analysis of time to event outcomes , 2005, Statistics in medicine.

[15]  Nicky J Welton,et al.  Estimation and adjustment of bias in randomized evidence by using mixed treatment comparison meta‐analysis , 2010 .

[16]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[17]  J. Sterne,et al.  Meta‒Analysis in Stata™ , 2008 .

[18]  Michele Tarsilla Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[19]  Simon G Thompson,et al.  Why and how Sources of Heterogeneity Should be Investigated , 2008 .

[20]  S G Thompson,et al.  Investigating underlying risk as a source of heterogeneity in meta-analysis. , 1997, Statistics in medicine.

[21]  Richard D Riley,et al.  Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. , 2007, Journal of clinical epidemiology.

[22]  A Whitehead,et al.  Meta‐analysis of ordinal outcomes using individual patient data , 2001, Statistics in medicine.

[23]  Douglas G. Altman,et al.  Models for potentially biased evidence in meta‐analysis using empirically based priors , 2009 .

[24]  Richard D Riley,et al.  Meta‐analysis of a binary outcome using individual participant data and aggregate data , 2010, Research synthesis methods.

[25]  C. Fidler,et al.  Mixed treatment comparison meta-analysis evaluating the relative efficacy of nucleos(t)ides for treatment of nucleos(t)ide-naive patients with chronic hepatitis B. , 2010, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[26]  George Davey Smith,et al.  Misleading meta-analysis , 1995, BMJ.

[27]  A. Marson,et al.  Multiple treatment comparisons in epilepsy monotherapy trials , 2007, Trials.

[28]  D J Spiegelhalter,et al.  Approximate cross‐validatory predictive checks in disease mapping models , 2003, Statistics in medicine.

[29]  L. Stewart,et al.  Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. , 1995, Statistics in medicine.

[30]  Nicky J Welton,et al.  Linear inference for mixed treatment comparison meta‐analysis: A two‐stage approach , 2011, Research synthesis methods.

[31]  C S Berkey,et al.  A random-effects regression model for meta-analysis. , 1995, Statistics in medicine.

[32]  M. Pittler Systematic Reviews in Health Care: Meta‐analysis in Context , 2010 .

[33]  David J Spiegelhalter,et al.  Bias modelling in evidence synthesis , 2009, Journal of the Royal Statistical Society. Series A,.

[34]  Douglas G. Altman,et al.  Systematic Reviews in Health Care: Meta-Analysis in Context: Second Edition , 2008 .

[35]  P. Emery,et al.  Certolizumab pegol for the treatment of rheumatoid arthritis , 2012, Expert opinion on biological therapy.

[36]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[37]  A J Sutton,et al.  Meta‐analysis of individual‐ and aggregate‐level data , 2008, Statistics in medicine.

[38]  M Egger,et al.  Intravenous magnesium for acute myocardial infarction. , 2007, The Cochrane database of systematic reviews.

[39]  S G Thompson,et al.  Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated , 1994, BMJ.

[40]  Deborah M Caldwell,et al.  NICE DSU Technical Support Document 4: Inconsistency in Networks of Evidence Based on Randomised Controlled Trials. , 2011 .

[41]  H. Goldstein,et al.  Meta‐analysis using multilevel models with an application to the study of class size effects , 2000 .

[42]  P C Lambert,et al.  An evaluation of bivariate random‐effects meta‐analysis for the joint synthesis of two correlated outcomes , 2007, Statistics in medicine.

[43]  Sofia Dias,et al.  Extending methods for investigating the relationship between treatment effect and baseline risk from pairwise meta‐analysis to network meta‐analysis , 2013, Statistics in medicine.

[44]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making Inconsistency in Networks of Evidence Based on Randomized Controlled Trials. , 2013 .

[45]  M C Simmonds,et al.  Covariate heterogeneity in meta‐analysis: Criteria for deciding between meta‐regression and individual patient data , 2007, Statistics in medicine.

[46]  Nicky J Welton,et al.  Study designs to detect sponsorship and other biases in systematic reviews. , 2010, Journal of clinical epidemiology.

[47]  David J Spiegelhalter,et al.  Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. , 2002, International journal of epidemiology.

[48]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making 2 , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[49]  Nicky J Welton,et al.  NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials , 2011 .

[50]  D G Altman,et al.  Indirect comparisons of competing interventions. , 2005, Health technology assessment.

[51]  Kurex Sidik,et al.  A comparison of heterogeneity variance estimators in combining results of studies , 2007, Statistics in medicine.

[52]  M. A. Best Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[53]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[54]  Alex J. Sutton,et al.  Evidence Synthesis for Decision Making 5: The Baseline Natural History Model. (5), , 2013 .

[55]  R. Riley,et al.  Meta-analysis of individual participant data: rationale, conduct, and reporting , 2010, BMJ : British Medical Journal.

[56]  Douglas G Altman,et al.  Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study , 2008, BMJ : British Medical Journal.

[57]  U. Ekelund,et al.  A proposed method of bias adjustment for meta-analyses of published observational studies , 2010, International journal of epidemiology.

[58]  Nicky J Welton,et al.  A cost-effectiveness analysis of genetic testing of the DRD2 Taq1A polymorphism to aid treatment choice for smoking cessation. , 2008, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[59]  Santiago G. Moreno,et al.  Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study , 2009, BMC medical research methodology.

[60]  Howard Raiffa,et al.  Applied Statistical Decision Theory. , 1961 .

[61]  Julian P T Higgins,et al.  Controlling the risk of spurious findings from meta‐regression , 2004, Statistics in medicine.

[62]  Jason Madan,et al.  Consistency between direct and indirect trial evidence: is direct evidence always more reliable? , 2011, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[63]  A Whitehead,et al.  Meta‐analysis of continuous outcome data from individual patients , 2001, Statistics in medicine.

[64]  M. McIntosh,et al.  The population risk as an explanatory variable in research synthesis of clinical trials. , 1996, Statistics in medicine.

[65]  Harold I Feldman,et al.  Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head , 2002, Statistics in medicine.

[66]  S Dias,et al.  Checking consistency in mixed treatment comparison meta‐analysis , 2010, Statistics in medicine.

[67]  N. Welton,et al.  Addressing between‐study heterogeneity and inconsistency in mixed treatment comparisons: Application to stroke prevention treatments in individuals with non‐rheumatic atrial fibrillation , 2009, Statistics in medicine.

[68]  Nicky J Welton,et al.  Controlling ecological bias in evidence synthesis of trials reporting on collapsed and overlapping covariate categories , 2010, Statistics in medicine.

[69]  Nicky J Welton,et al.  Evidence Synthesis for Decision Making in Healthcare , 2012 .

[70]  Catrin Tudur Smith,et al.  Assessing the consistency assumption by exploring treatment by covariate interactions in mixed treatment comparison meta‐analysis: individual patient‐level covariates versus aggregate trial‐level covariates , 2012, Statistics in medicine.

[71]  A E Ades,et al.  Correction: Interpretation of Random Effects Meta-analysis in Decision Models , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[72]  David J Spiegelhalter,et al.  A re-evaluation of random-effects meta-analysis , 2009, Journal of the Royal Statistical Society. Series A,.

[73]  N. Draper,et al.  Applied Regression Analysis. , 1967 .

[74]  Richard D Riley,et al.  Meta‐analysis of continuous outcomes combining individual patient data and aggregate data , 2008, Statistics in medicine.

[75]  A E Ades,et al.  The Interpretation of Random-Effects Meta-Analysis in Decision Models , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[76]  Gerta Rücker,et al.  Bmc Medical Research Methodology Open Access Undue Reliance on I 2 in Assessing Heterogeneity May Mislead , 2022 .

[77]  Nicola J Cooper,et al.  Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications , 2009, BMJ : British Medical Journal.

[78]  Alan Brennan,et al.  Using mixed treatment comparisons and meta‐regression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis , 2007, Statistics in Medicine.

[79]  Nicky J Welton,et al.  NICE DSU TECHNICAL SUPPORT DOCUMENT 5: EVIDENCE SYNTHESIS IN THE BASELINE NATURAL HISTORY MODEL , 2011 .

[80]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[81]  D. Solomon,et al.  Applied Statistical Decision Theory. , 1961 .