Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

Objective To investigate the agreement between direct and indirect comparisons of competing healthcare interventions. Design Meta-epidemiological study based on sample of meta-analyses of randomised controlled trials. Data sources Cochrane Database of Systematic Reviews and PubMed. Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic. Main outcome measure Inconsistency measured by the difference in the log odds ratio between the direct and indirect methods. Results The study included 112 independent trial networks (including 1552 trials with 478 775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined. Conclusions Significant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence.

[1]  D G Altman,et al.  Indirect comparison in evaluating relative efficacy illustrated by antimicrobial prophylaxis in colorectal surgery. , 2000, Controlled clinical trials.

[2]  G. Lu,et al.  Assessing Evidence Inconsistency in Mixed Treatment Comparisons , 2006 .

[3]  J Chilcott,et al.  Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review. , 2010, Health technology assessment.

[4]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[5]  T. Lumley.  Network meta‐analysis for indirect treatment comparisons , 2002, Statistics in medicine.

[6]  Douglas G Altman,et al.  Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses , 2003, BMJ : British Medical Journal.

[7]  G. Lu,et al.  Combination of direct and indirect evidence in mixed treatment comparisons , 2004, Statistics in medicine.

[8]  B. Elewski,et al.  A comparison of 15% azelaic acid gel and 0.75% metronidazole gel in the topical treatment of papulopustular rosacea: results of a randomized trial. , 2003, Archives of dermatology.

[9]  Paula Williamson,et al.  Indirect Comparisons: A Review of Reporting and Methodological Quality , 2010, PloS one.

[10]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[11]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[12]  Nicky J Welton,et al.  Evaluating novel agent effects in multiple‐treatments meta‐regression , 2010, Statistics in medicine.

[13]  S D Walter,et al.  The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. , 1997, Journal of clinical epidemiology.

[14]  Douglas G Altman,et al.  Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews , 2009, BMJ : British Medical Journal.

[15]  D G Altman,et al.  Indirect comparisons of competing interventions. , 2005, Health technology assessment.

[16]  D. Henson,et al.  The effectiveness of the Heidelberg Retina Tomograph and laser diagnostic glaucoma scanning system (GDx) in detecting and monitoring glaucoma. , 2005, Health technology assessment.

[17]  Gerald Gartlehner,et al.  Direct versus indirect comparisons: A summary of the evidence , 2008, International Journal of Technology Assessment in Health Care.

[18]  Douglas G Altman,et al.  Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study , 2008, BMJ : British Medical Journal.

[19]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[20]  I Harvey,et al.  Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. , 2008, Journal of clinical epidemiology.

[21]  S Wordsworth,et al.  Indirect comparisons of treatments based on systematic reviews of randomised controlled trials , 2009, International journal of clinical practice.

[22]  Deborah M Caldwell,et al.  Simultaneous comparison of multiple treatments: combining direct and indirect evidence , 2005, BMJ : British Medical Journal.

[23]  Jonathan J Deeks,et al.  Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomes , 2002, Statistics in medicine.

[24]  S Dias,et al.  Checking consistency in mixed treatment comparison meta‐analysis , 2010, Statistics in medicine.

[25]  Andrew R Willan,et al.  Indirect comparison: relative risk fallacies and odds solution. , 2009, Journal of clinical epidemiology.