Overall similarity and consistency assessment scores are not sufficiently accurate for predicting discrepancy between direct and indirect comparison estimates

Objectives Indirect comparison methods have been increasingly used to assess the effectiveness of different interventions comparatively. This study evaluated a Trial Similarity and Evidence Consistency Assessment (TSECA) framework for assessing key assumptions underlying the validity of indirect comparisons. Study Design and Setting We applied the TSECA framework to 94 Cochrane Systematic Reviews that provided data to compare two interventions by both direct and indirect comparisons. Using the TSECA framework, two reviewers independently assessed and scored trial similarity and evidence consistency. A detailed case study provided further insight into the usefulness and limitations of the framework proposed. Results Trial similarity and evidence consistency scores obtained using the assessment framework were not associated with statistically significant inconsistency between direct and indirect estimates. The case study illustrated that the assessment framework could be used to identify potentially important differences in participants, interventions, and outcome measures between different sets of trials in the indirect comparison. Conclusion Although the overall trial similarity and evidence consistency scores are unlikely to be sufficiently accurate for predicting inconsistency between direct and indirect estimates, the assessment framework proposed in this study can be a useful tool for identifying between-trial differences that may threaten the validity of indirect treatment comparisons.

[1]  S D Walter,et al.  The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. , 1997, Journal of clinical epidemiology.

[2]  A. Beckett,et al.  AKUFO AND IBARAPA. , 1965, Lancet.

[3]  Douglas G Altman,et al.  Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews , 2009, BMJ : British Medical Journal.

[4]  D. Altman,et al.  Assessing Risk of Bias in Included Studies , 2008 .

[5]  Alan Brennan,et al.  Using mixed treatment comparisons and meta‐regression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis , 2007, Statistics in Medicine.

[6]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[7]  Roger Chou,et al.  Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses , 2006, The Lancet.

[8]  I Harvey,et al.  Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. , 2008, Journal of clinical epidemiology.

[9]  A Whitehead,et al.  Borrowing strength from external trials in a meta-analysis. , 1996, Statistics in medicine.

[10]  Deborah M Caldwell,et al.  Simultaneous comparison of multiple treatments: combining direct and indirect evidence , 2005, BMJ : British Medical Journal.

[11]  M. Graber,et al.  Interventions for rosacea. , 2005, The Cochrane database of systematic reviews.

[12]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[13]  Douglas G. Altman,et al.  Chapter 9: Analysing Data and Undertaking Meta-Analyses , 2008 .

[14]  B. Elewski,et al.  A comparison of 15% azelaic acid gel and 0.75% metronidazole gel in the topical treatment of papulopustular rosacea: results of a randomized trial. , 2003, Archives of dermatology.

[15]  Douglas G Altman,et al.  Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses , 2003, BMJ : British Medical Journal.

[16]  Joseph C Cappelleri,et al.  Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. , 2011, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[17]  Julian P T Higgins,et al.  A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. , 2009, Journal of clinical epidemiology.

[18]  Georgia Salanti,et al.  Directed acyclic graphs can help understand bias in indirect and mixed treatment comparisons. , 2012, Journal of clinical epidemiology.

[19]  G. Lu,et al.  Combination of direct and indirect evidence in mixed treatment comparisons , 2004, Statistics in medicine.

[20]  M. Coory,et al.  Frequency of Treatment-EffectModification Affecting Indirect Comparisons , 2010, PharmacoEconomics.

[21]  P. Rothwell,et al.  Factors That Can Affect the External Validity of Randomised Controlled Trials , 2006, PLoS clinical trials.

[22]  R. Matthews,et al.  What are the implications of optimism bias in clinical research? , 2006, The Lancet.

[23]  N. Welton,et al.  Addressing between‐study heterogeneity and inconsistency in mixed treatment comparisons: Application to stroke prevention treatments in individuals with non‐rheumatic atrial fibrillation , 2009, Statistics in medicine.

[24]  Paula Williamson,et al.  Indirect Comparisons: A Review of Reporting and Methodological Quality , 2010, PloS one.

[25]  Alex J Sutton,et al.  Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study , 2011, BMJ : British Medical Journal.

[26]  Tianjing Li,et al.  Network meta-analysis-highly attractive but more methodological research is needed , 2011, BMC medicine.

[27]  Deborah M Caldwell,et al.  NICE DSU Technical Support Document 4: Inconsistency in Networks of Evidence Based on Randomised Controlled Trials. , 2011 .

[28]  D. Altman,et al.  Chapter 8: Assessing risk of bias in included studies , 2008 .

[29]  Ellen Frank,et al.  Moderators of treatment outcomes: clinical, research, and policy importance. , 2006, JAMA.

[30]  T. Lumley.  Network meta‐analysis for indirect treatment comparisons , 2002, Statistics in medicine.