The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews

Objective To examine the prevalence of outcome reporting bias—the selection for publication of a subset of the original recorded outcome variables on the basis of the results—and its impact on Cochrane reviews. Design A nine point classification system for missing outcome data in randomised trials was developed and applied to the trials assessed in a large, unselected cohort of Cochrane systematic reviews. Researchers who conducted the trials were contacted and the reason sought for the non-reporting of data. A sensitivity analysis was undertaken to assess the impact of outcome reporting bias on reviews that included a single meta-analysis of the review primary outcome. Results More than half (157/283 (55%)) the reviews did not include full data for the review primary outcome of interest from all eligible trials. The median amount of review outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. It was clear from the publications for 155 (6%) of the 2486 assessable trials that the researchers had measured and analysed the review primary outcome but did not report or only partially reported the results. For reports that did not mention the review primary outcome, our classification regarding the presence of outcome reporting bias was shown to have a sensitivity of 88% (95% CI 65% to 100%) and specificity of 80% (95% CI 69% to 90%) on the basis of responses from 62 trialists. A third of Cochrane reviews (96/283 (34%)) contained at least one trial with high suspicion of outcome reporting bias for the review primary outcome. In a sensitivity analysis undertaken for 81 reviews with a single meta-analysis of the primary outcome of interest, the treatment effect estimate was reduced by 20% or more in 19 (23%). Of the 42 meta-analyses with a statistically significant result only, eight (19%) became non-significant after adjustment for outcome reporting bias and 11 (26%) would have overestimated the treatment effect by 20% or more. Conclusions Outcome reporting bias is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews. Individuals conducting systematic reviews need to address explicitly the issue of missing outcome data for their review to be considered a reliable source of evidence. Extra care is required during data extraction, reviewers should identify when a trial reports that an outcome was measured but no results were reported or events observed, and contact with trialists should be encouraged.

[1]  D. W. Lewis,et al.  Diagnostic tests , 1995, Journal.

[2]  M. Smithson.  Statistics with confidence , 2000 .

[3]  P. Williamson,et al.  Bias in meta‐analysis due to outcome variable selection within studies , 2000 .

[4]  J L Hutton,et al.  Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies. , 2000, Statistics in medicine.

[5]  P. Tugwell,et al.  A Comparison of the Quality of Cochrane Reviews and Systematic Reviews Published in Paper-Based Journals , 2002, Evaluation & the health professions.

[6]  Dan Jackson,et al.  A Bound for Publication Bias Based on the Fraction of Unpublished Studies , 2004, Biometrics.

[7]  D. Altman,et al.  Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors , 2005, BMJ : British Medical Journal.

[8]  C. Gamble,et al.  Identification and impact of outcome selection bias in meta‐analysis , 2005, Statistics in medicine.

[9]  A Eisinga,et al.  Compression stockings for preventing deep vein thrombosis in airline passengers. , 2006, The Cochrane database of systematic reviews.

[10]  C. Gamble,et al.  Application and investigation of a bound for outcome reporting bias , 2007, Trials.

[11]  K. Shojania,et al.  Opiates and Acute Abdominal Pain—Reply , 2007 .

[12]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews , 2007, PLoS medicine.

[13]  G. Guyatt,et al.  Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. , 2007, JAMA.

[14]  I. Sim,et al.  Reporting the findings of clinical trials: a discussion paper. , 2008, Bulletin of the World Health Organization.

[15]  M. Clarke.  Standardising Outcomes in Paediatric Clinical Trials , 2008, PLoS Medicine.

[16]  D. Altman,et al.  Reporting of clinical trials: a review of research funders' guidelines , 2008, Trials.

[17]  Paula R Williamson,et al.  A Systematic Review of Studies That Aim to Determine Which Outcomes to Measure in Clinical Trials in Children , 2008, PLoS medicine.

[18]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[19]  Jane Fridlyand,et al.  Academic Editor: Jonathan Rees, University of Edinburgh, United Kingdom , 2007 .

[20]  Nicola J Cooper,et al.  Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications , 2009, BMJ : British Medical Journal.

[21]  M. Pittler.  Systematic Reviews in Health Care: Meta‐analysis in Context , 2010 .

[22]  P. Alderson,et al.  Human albumin solution for resuscitation and volume expansion in critically ill patients. , 2011, The Cochrane database of systematic reviews.