Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies

BackgroundThe validity of research synthesis is threatened if published studies comprise a biased selection of all studies that have been conducted. We conducted a meta-analysis to ascertain the strength and consistency of the association between study results and formal publication.MethodsThe Cochrane Methodology Register Database, MEDLINE and other electronic bibliographic databases were searched (to May 2009) to identify empirical studies that tracked a cohort of studies and reported the odds of formal publication by study results. Reference lists of retrieved articles were also examined for relevant studies. Odds ratios were used to measure the association between formal publication and significant or positive results. Included studies were separated into subgroups according to starting time of follow-up, and results from individual cohort studies within the subgroups were quantitatively pooled.ResultsWe identified 12 cohort studies that followed up research from inception, four that included trials submitted to a regulatory authority, 28 that assessed the fate of studies presented as conference abstracts, and four cohort studies that followed manuscripts submitted to journals. The pooled odds ratio of publication of studies with positive results, compared to those without positive results (publication bias) was 2.78 (95% CI: 2.10 to 3.69) in cohorts that followed from inception, 5.00 (95% CI: 2.01 to 12.45) in trials submitted to regulatory authority, 1.70 (95% CI: 1.44 to 2.02) in abstract cohorts, and 1.06 (95% CI: 0.80 to 1.39) in cohorts of manuscripts.ConclusionDissemination of research findings is likely to be a biased process. Publication bias appears to occur early, mainly before the presentation of findings at conferences or submission of manuscripts to journals.

[1]  A J Sutton,et al.  Publication and related biases. , 2000, Health technology assessment.

[2]  L. Stewart,et al.  Time to publication for results of clinical trials. , 2007, The Cochrane database of systematic reviews.

[3]  Ian F Tannock,et al.  Factors associated with failure to publish large randomized trials presented at an oncology meeting. , 2003, JAMA.

[4]  S. Hahn,et al.  Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. , 2002, Journal of evaluation in clinical practice.

[5]  T. Sheldon,et al.  Factors influencing the publication of health research , 2004, International Journal of Technology Assessment in Health Care.

[6]  R. Simes,et al.  Publication bias: evidence of delayed publication in a cohort study of clinical research projects , 1997, BMJ.

[7]  E. Decullier,et al.  Impact of funding on biomedical research: a retrospective cohort study , 2006, BMC public health.

[8]  J. Evers,et al.  Publication bias in reproductive research. , 2000, Human reproduction.

[9]  I. Tannock,et al.  The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. , 1992, Annals of oncology : official journal of the European Society for Medical Oncology.

[10]  J. Ioannidis.  Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. , 1998, JAMA.

[11]  Jane M. Young,et al.  Publication Bias in Abstracts Presented to the Annual Meeting of the American Academy of Orthopaedic Surgeons , 2007, Journal of orthopaedic surgery.

[12]  H. Dubben,et al.  Systematic review of publication bias in studies on publication bias , 2005, BMJ : British Medical Journal.

[13]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[14]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[15]  A. Laupacis.  Methodological studies of systematic reviews: is there publication bias? , 1997, Archives of internal medicine.

[16]  B. Druss,et al.  Tracking publication outcomes of National Institutes of Health grants. , 2005, The American journal of medicine.

[17]  Role of a research ethics committee in follow-up and publication of results , 2003, The Lancet.

[18]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[19]  V. Landry.  The publication outcome for the papers presented at the 1990 ABA conference. , 1996, The Journal of burn care & rehabilitation.

[20]  D. Provenzale,et al.  Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form. , 2001, Gastrointestinal endoscopy.

[21]  Veröffentlichungen von Dissertationen zu unkonventionellen medizinischen Therapie- und Diagnoseverfahren – ein Beitrag zum «publication bias» , 2000, Complementary Medicine Research.

[22]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[23]  H. Cooper,et al.  Finding the Missing Science : The Fate of Studies Submitted for Review by a Human Subjects Committee , 2004 .

[24]  T P Klassen,et al.  The inclusion of reports of randomised trials published in languages other than English in systematic reviews. , 2003, Health technology assessment.

[25]  B. Beermann,et al.  Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications , 2003, BMJ : British Medical Journal.

[26]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[27]  Sally Hopewell,et al.  Publication bias in clinical trials due to statistical significance or direction of trial results. , 2009, The Cochrane database of systematic reviews.

[28]  P. Hashkes,et al.  The publication rate of abstracts from the 4th Park City Pediatric Rheumatology meeting in peer-reviewed journals: what factors influenced publication? , 2003, The Journal of rheumatology.

[29]  D. Altman,et al.  Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors , 2005, BMJ : British Medical Journal.

[30]  S. Halpern,et al.  PUBLISHED Meeting Abstracts IN OBSTETRICAL ANESTHESIA: FULL PUBLICATION RATES AND DATA RELIABILITY A69 , 2001 .

[31]  J. Windeler,et al.  [Publications of dissertations on unconventional medical therapy and diagnosis procedures--a contribution to "publication bias"]. , 2000, Forschende Komplementarmedizin und klassische Naturheilkunde = Research in complementary and natural classical medicine.

[32]  M. Petticrew,et al.  Lost information? The fate of papers presented at the 40th society for Social Medicine Conference. , 1999, Journal of epidemiology and community health.

[33]  K. J. Lim,et al.  Publication Rates for Abstracts Presented by Korean Investigators at Major Radiology Meetings , 2008, Korean journal of radiology.

[34]  J. Sterne,et al.  The publication process itself was the major cause of publication bias in genetic epidemiology. , 2006, Journal of clinical epidemiology.

[35]  R. Wears,et al.  Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. , 1998, JAMA.

[36]  E. von Elm,et al.  Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. , 2008, Swiss medical weekly.

[37]  F. Wolf,et al.  Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. , 2007, The Journal of bone and joint surgery. American volume.

[38]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[39]  B. Shakiba,et al.  Transition from congress abstract to full publication for clinical trials presented at laser meetings , 2008, Lasers in Medical Science.

[40]  D. Moher,et al.  Abstracts of randomized controlled trials presented at the society for pediatric research meeting: an example of publication bias. , 2002, Archives of pediatrics & adolescent medicine.

[41]  C. Begg,et al.  Publication bias and dissemination of clinical research. , 1989, Journal of the National Cancer Institute.

[42]  Ida Sim,et al.  Publication of Clinical Trials Supporting Successful New Drug Applications: A Literature Analysis , 2008, PLoS medicine.

[43]  F. Song.  Review of publication bias in studies on publication bias: Studies on publication bias are probably susceptible to the bias they study , 2005, BMJ : British Medical Journal.

[44]  J. Castillo,et al.  Fate of abstracts from the Paris 1995 European Society of Anaesthesiologists meeting , 2002, European journal of anaesthesiology.

[45]  Lisa Bero,et al.  Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation , 2008, PLoS medicine.

[46]  G K Kiroff,et al.  Publication bias in presentations o the Annual Scientific Congress , 2001, ANZ journal of surgery.

[47]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[48]  G. Knoll,et al.  Factors Associated with Publication Following Presentation at a Transplantation Meeting , 2006, American journal of transplantation : official journal of the American Society of Transplantation and the American Society of Transplant Surgeons.

[49]  P. Sandercock,et al.  No evidence of bias in the process of publication of diagnostic accuracy studies in stroke submitted as abstracts. , 2009, Journal of clinical epidemiology.

[50]  Jay Brand,et al.  File Drawer Problem , 2022, The SAGE Encyclopedia of Research Design.

[51]  David Hailey,et al.  BMC Medical Research Methodology BioMed Central BMC 22002, Medical Research Methodology , 2001 .

[52]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[53]  D. Rennie,et al.  Publication bias in editorial decision making. , 2002, JAMA.

[54]  Jeffrey L Saver,et al.  Frequency and Determinants of Nonpublication of Research in the Stroke Literature , 2006, Stroke.

[55]  L. Bero,et al.  Publication bias and research on passive smoking: comparison of published and unpublished studies. , 1998, JAMA.

[56]  David Sohn.  Publication bias and the evaluation of psychotherapy efficacy in reviews of the research literature , 1996 .

[57]  R. Rosenthal,et al.  Selective publication of antidepressant trials and its influence on apparent efficacy. , 2008, The New England journal of medicine.

[58]  W. Abuznadah,et al.  Research publication in pediatric surgery: a cross-sectional study of papers presented at the Canadian Association of Pediatric Surgeons and the American Pediatric Surgery Association. , 2006, Journal of pediatric surgery.

[59]  Gerta Rücker,et al.  Bmc Medical Research Methodology Open Access Undue Reliance on I 2 in Assessing Heterogeneity May Mislead , 2022 .

[60]  D. Altman,et al.  Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research , 2004, Canadian Medical Association Journal.

[61]  K. Dickersin,et al.  NIH clinical trials and publication bias. , 1993, The Online journal of current clinical trials.

[62]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[63]  J. Holroyd-Leduc,et al.  Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals , 2006, The Medical journal of Australia.

[64]  M. Bhandari,et al.  Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). , 2008, The Journal of bone and joint surgery. American volume.

[65]  C. Begg,et al.  Publication bias : a problem in interpreting medical data , 1988 .

[66]  G. Antes,et al.  Clinical research projects at a German medical faculty: follow-up from ethical approval to publication and citation by others , 2008, Journal of Medical Ethics.

[67]  K. Dickersin.  How important is publication bias? A synthesis of available data. , 1997, AIDS education and prevention : official publication of the International Society for AIDS Education.

[68]  E. von Elm,et al.  Full publication of results initially presented in abstracts. , 2007, The Cochrane database of systematic reviews.

[69]  P. Dahm,et al.  Factors associated with the full publication of studies presented in abstract form at the annual meeting of the American Urological Association. , 2007, The Journal of urology.

[70]  E. Decullier,et al.  Fate of biomedical research protocols and publication bias in France: retrospective cohort study , 2005, BMJ : British Medical Journal.

[71]  K. Dickersin,et al.  Full publication of results initially presented in abstracts. A meta-analysis. , 1994 .

[72]  A. Bardy.  Bias in reporting clinical trials. , 1998, British journal of clinical pharmacology.

[73]  J. Sterne,et al.  How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. , 2003, Health technology assessment.

[74]  Richard M Rosenfeld,et al.  Factors influencing publication of abstracts presented at the AAO-HNS Annual Meeting , 2006, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[75]  S. Golder,et al.  The effectiveness and cost-effectiveness of prophylactic removal of wisdom teeth. , 2000, Health technology assessment.

[76]  Jane M. Young,et al.  PUBLICATION BIAS IN PAPERS PRESENTED TO THE AUSTRALIAN ORTHOPAEDIC ASSOCIATION ANNUAL SCIENTIFIC MEETING , 2006, ANZ journal of surgery.

[77]  A. Sutton,et al.  Comparison of two methods to detect publication bias in meta-analysis. , 2006, JAMA.

[78]  Kay Dickersin,et al.  What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion , 2007, BMC medical research methodology.

[79]  D. Ashby,et al.  Time to publication as full reports of abstracts of randomized controlled trials in cystic fibrosis , 1998, Pediatric pulmonology.

[80]  Alex J. Sutton,et al.  Publication and related biases: a review , 2000 .

[81]  K. Dickersin.  The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.