Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses

Abstract Objective: To determine the validity of adjusted indirect comparisons by using data from published meta-analyses of randomised trials. Design: Direct comparison of different interventions in randomised trials and adjusted indirect comparison in which two interventions were compared through their relative effect versus a common comparator. The discrepancy between the direct and adjusted indirect comparison was measured by the difference between the two estimates. Data sources: Database of abstracts of reviews of effectiveness (1994-8), the Cochrane database of systematic reviews, Medline, and references of retrieved articles. Results: 44 published meta-analyses (from 28 systematic reviews) provided sufficient data. In most cases, results of adjusted indirect comparisons were not significantly different from those of direct comparisons. A significant discrepancy (P<0.05) was observed in three of the 44 comparisons between the direct and the adjusted indirect estimates. There was a moderate agreement between the statistical conclusions from the direct and adjusted indirect comparisons (κ 0.51). The direction of discrepancy between the two estimates was inconsistent. Conclusions: Adjusted indirect comparisons usually but not always agree with the results of head to head randomised trials. When there is no or insufficient direct evidence from randomised trials, the adjusted indirect comparison may provide useful or supplementary information on the relative efficacy of competing interventions. The validity of the adjusted indirect comparisons depends on the internal validity and similarity of the included trials. What is already known on this topic Many competing interventions have not been compared in randomised trials Indirect comparison of competing interventions has been carried out in systematic reviews, often implicitly Indirect comparison adjusted by a common control can partially take account of prognostic characteristics of patients in different trials What this study adds Results of adjusted indirect comparison usually, but not always, agree with those of head to head randomised trials The validity of adjusted indirect comparisons depends on the internal validity and similarity of the trials involved

[1]  D G Altman,et al.  Statistics notes: Absence of evidence is not evidence of absence , 1995 .

[2]  U. Abel,et al.  The role of randomization in clinical studies: myths and beliefs. , 1999, Journal of clinical epidemiology.

[3]  L. Fisher,et al.  Active-control trials: how would a new agent compare with placebo? A method illustrated with clopidogrel, aspirin, and placebo. , 2001, American heart journal.

[4]  S D Walter,et al.  The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. , 1997, Journal of clinical epidemiology.

[5]  M. Palmer.  Clinical Trials: A Practical Approach , 1985 .

[6]  D L Sackett,et al.  Users' Guides to the Medical Literature: XIX. Applying clinical trial results B. Guidelines for determining whether a drug is exerting (more than) a class effect. , 1999, JAMA.

[7]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[8]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[9]  D G Altman,et al.  Indirect comparison in evaluating relative efficacy illustrated by antimicrobial prophylaxis in colorectal surgery. , 2000, Controlled clinical trials.

[10]  Alex J. Sutton,et al.  Publication and related biases: a review , 2000 .

[11]  A R Feinstein,et al.  Claims of Equivalence in Medical Research: Are They Supported by the Evidence? , 2000, Annals of Internal Medicine.

[12]  Douglas G. Altman,et al.  Practical statistics for medical research , 1990 .

[13]  C H Schmid,et al.  Large trials vs meta-analysis of smaller trials : How do their results compare ? , 1996 .

[14]  A Whitehead,et al.  Borrowing strength from external trials in a meta-analysis. , 1996, Statistics in medicine.

[15]  G. Guyatt,et al.  Users' guides to the medical literature. , 1993, JAMA.

[16]  N. Black.  Why we need observational studies to evaluate the effectiveness of health care , 1996, BMJ.

[17]  B. T. Johnson,et al.  Large trials vs meta-analysis of smaller trials. , 1997, JAMA.

[18]  A J Sutton,et al.  Publication and related biases. , 2000, Health technology assessment.

[19]  David F. Kong,et al.  Statistical Methods for Comparison to Placebo in Active-Control Trials , 2001 .