Reporting bias and other biases affecting systematic reviews and meta-analyses: a methodological commentary

Systematic reviews and meta-analyses often occupy the top of the hierarchy of evidence in support of evidence-based clinical practice. These studies commonly inform the formulation of clinical guidelines. Bias can intrude at several levels during the conduct of systematic reviews. The effect these various biases, in particular reporting bias, have on pooled estimates and review inferences are potentially significant. In this review, we describe several forms of selection and reporting biases that may occur during the conduct of a systematic review, how these biases might affect a review and what steps could help minimize their influence on review inferences. Specifically, we support calls for prospective international trial registration and open access to trial protocols as two potential solutions that may improve the methodological quality of systematic reviews and the validity of their results.

[1]  Gunther Eysenbach,et al.  Report of a case of cyberplagiarism - and reflections on detecting and preventing academic misconduct using the Internet , 2000, Journal of medical Internet research.

[2]  Jesse A Berlin,et al.  Searching one or two databases was insufficient for meta-analysis of observational studies. , 2005, Journal of clinical epidemiology.

[3]  D. Altman,et al.  Outcome selection bias in meta-analysis , 2005, Statistical methods in medical research.

[4]  C. van Weel,et al.  Alpha-glucosidase inhibitors for patients with type 2 diabetes: results from a Cochrane systematic review and meta-analysis. , 2005, Diabetes care.

[5]  T. Greenhalgh,et al.  Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources , 2005, BMJ : British Medical Journal.

[6]  I. Olkin,et al.  The case of the misleading funnel plot , 2006, BMJ : British Medical Journal.

[7]  M. Egger,et al.  Bias in location and selection of studies. , 1998, BMJ.

[8]  B. Bauer,et al.  Cochrane systematic reviews in acupuncture: methodological diversity in database searching. , 2005, Journal of alternative and complementary medicine.

[9]  D. Altman,et al.  Waiving confidentiality for the greater good , 2006, BMJ : British Medical Journal.

[10]  F. Mosteller,et al.  A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. , 1992, JAMA.

[11]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[12]  W. McIlroy,et al.  Should unpublished data be included in meta-analyses? Current convictions and controversies. , 1993, JAMA.

[13]  L. Bero,et al.  Publication bias and research on passive smoking: comparison of published and unpublished studies. , 1998, JAMA.

[14]  R. Horton,et al.  WHO facilitates international collaboration in setting standards for clinical trial registration , 2005, The Lancet.

[15]  David R. Jones,et al.  Empirical assessment of effect of publication bias on meta-analyses , 2000, BMJ : British Medical Journal.

[16]  K. Dickersin,et al.  Systematic Reviews: Identifying relevant studies for systematic reviews , 1994 .

[17]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[18]  M. Clarke,et al.  Identifying relevant studies for systematic reviews , 1995, BMJ.

[19]  K. Dickersin,et al.  NIH clinical trials and publication bias. , 1993, The Online journal of current clinical trials.

[20]  Peter Fonagy,et al.  Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data , 2004, The Lancet.

[21]  Emil H Schemitsch,et al.  Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. , 2004, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[22]  J. Copas,et al.  Meta-analysis, funnel plots and sensitivity analysis. , 2000, Biostatistics.

[23]  Christopher H Schmid,et al.  In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. , 2005, Journal of clinical epidemiology.

[24]  R. Simes,et al.  Publication bias: evidence of delayed publication in a cohort study of clinical research projects , 1997, BMJ.

[25]  E. Wager.  The need for trial identifiers , 2004, Current medical research and opinion.

[26]  Erik von Elm,et al.  Different patterns of duplicate publication: an analysis of articles used in systematic reviews. , 2004, JAMA.

[27]  David Moher,et al.  The Medical Review Article Revisited: Has the Science Improved? , 1999, Annals of Internal Medicine.

[28]  George Davey Smith,et al.  meta-analysis bias in location and selection of studies , 1998 .

[29]  Erland Erdmann,et al.  Secondary prevention of macrovascular events in patients with type 2 diabetes in the PROactive Study (PROspective pioglitAzone Clinical Trial In macroVascular Events): a randomised controlled trial , 2005, The Lancet.

[30]  G. Guyatt,et al.  Telecare for patients with type 1 diabetes and inadequate glycemic control: a randomized controlled trial and meta-analysis. , 2004, Diabetes care.

[31]  Ida Sim,et al.  Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1) , 2005, BMJ : British Medical Journal.

[32]  R. Deyo.  Industry support and correlation to study outcome for papers published in Spine: Point of view , 2005 .

[33]  G H Guyatt,et al.  Publication bias: a brief review for clinicians. , 2000, Mayo Clinic proceedings.

[34]  John Hoey,et al.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors. , 2005, Circulation.

[35]  Liza Gibson,et al.  WHO warns of a polio epidemic in Africa , 2004, BMJ : British Medical Journal.

[36]  P. Langhorne.  Bias in meta-analysis detected by a simple, graphical test. Prospectively identified trials could be used for comparison with meta-analyses. , 1998, BMJ.

[37]  L. Rubenstein,et al.  Bias in meta-analysis detected by a simple, graphical test. Test had 10% false positive rate. , 1998, BMJ.

[38]  P. Tugwell,et al.  Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? , 2000, The Lancet.

[39]  D. Cook,et al.  Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions , 1997, Annals of Internal Medicine.

[40]  D. Altman,et al.  Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors , 2005, BMJ : British Medical Journal.

[41]  C. Lengeler,et al.  Language bias in randomised controlled trials published in English and German , 1997, The Lancet.

[42]  W. Isley.  Pioglitazone did not reduce a composite endpoint of macrovascular complications and increased risk for heart failure in type 2 diabetes with macrovascular disease. , 2006, ACP journal club.

[43]  P. Glasziou,et al.  Bias in meta-analysis detected by a simple, graphical test. Graphical test is itself biased. , 1998, BMJ.

[44]  Jonathan A C Sterne,et al.  Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. , 2001, BMJ.

[45]  G. Antes,et al.  Randomised trials in German-language journals , 1996, The Lancet.

[46]  Xiaoyi Jiang,et al.  Redundant publications in scientific ophthalmologic journals: the tip of the iceberg? , 2004, Ophthalmology.

[47]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[48]  Ian F Tannock,et al.  Factors associated with failure to publish large randomized trials presented at an oncology meeting. , 2003, JAMA.

[49]  M. Hanefeld,et al.  Acarbose reduces the risk for myocardial infarction in type 2 diabetic patients: meta-analysis of seven long-term studies. , 2004, European heart journal.

[50]  C. Gamble,et al.  Identification and impact of outcome selection bias in meta‐analysis , 2005, Statistics in medicine.

[51]  E Frank,et al.  Authors' criteria for selecting journals. , 1994, JAMA.

[52]  J P Vandenbroucke,et al.  Bias in meta-analysis detected by a simple, graphical test , 1998 .

[53]  R. Wears,et al.  Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. , 1998, JAMA.

[54]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[55]  J. Grauer,et al.  Industry Support and Correlation to Study Outcome for Papers Published in Spine , 2005, Spine.

[56]  David Hailey,et al.  BMC Medical Research Methodology BioMed Central BMC 22002, Medical Research Methodology , 2001 .

[57]  C. Mulrow.  The medical review article: state of the science. , 1987, Annals of internal medicine.

[58]  L. Rubenstein,et al.  Bias in meta-analysis detected by a simple, graphical test. Asymmetry detected in funnel plot was probably due to true heterogeneity. , 1998, BMJ.