Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature

Background Preprint usage is growing rapidly in the life sciences; however, questions remain on the relative quality of preprints when compared to published articles. An objective dimension of quality that is readily measurable is completeness of reporting, as transparency can improve the reader’s ability to independently interpret data and reproduce findings. Methods In this observational study, we initially compared independent samples of articles published in bioRxiv and in PubMed-indexed journals in 2016 using a quality of reporting questionnaire. After that, we performed paired comparisons between preprints from bioRxiv to their own peer-reviewed versions in journals. Results Peer-reviewed articles had, on average, higher quality of reporting than preprints, although the difference was small, with absolute differences of 5.0% [95% CI 1.4, 8.6] and 4.7% [95% CI 2.4, 7.0] of reported items in the independent samples and paired sample comparison, respectively. There were larger differences favoring peer-reviewed articles in subjective ratings of how clearly titles and abstracts presented the main findings and how easy it was to locate relevant reporting information. Changes in reporting from preprints to peer-reviewed versions did not correlate with the impact factor of the publication venue or with the time lag from bioRxiv to journal publication. Conclusions Our results suggest that, on average, publication in a peer-reviewed journal is associated with improvement in quality of reporting. They also show that quality of reporting in preprints in the life sciences is within a similar range as that of peer-reviewed articles, albeit slightly lower on average, supporting the idea that preprints should be considered valid scientific contributions.

[1]  T. Jefferson,et al.  Editorial peer review for improving the quality of reports of biomedical studies. , 2007, The Cochrane database of systematic reviews.

[2]  D. Moher,et al.  A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals , 2019, BMC Medicine.

[3]  D. Altman,et al.  Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial , 2011, BMJ : British Medical Journal.

[4]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Journal of the American Podiatric Medical Association.

[5]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[6]  D. Moher,et al.  The EQUATOR Network and reporting guidelines: Helping to achieve high standards in reporting health research studies. , 2009, Maturitas.

[7]  Vincent Larivière,et al.  Gender and international diversity improves equity in peer review , 2018, bioRxiv.

[8]  Jean-Pierre EN Pierie,et al.  Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde , 1996, The Lancet.

[9]  S. Pocock,et al.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies , 2007, The Lancet.

[10]  Gillian L. Currie,et al.  Risk of Bias in Reports of In Vivo Research: A Focus for Improvement , 2015, PLoS biology.

[11]  Karin Wårdell,et al.  A physical action potential generator: design, implementation and evaluation , 2015, Front. Neurosci..

[12]  Adam Marcus,et al.  Publishing: The peer-review scam , 2014, Nature.

[13]  Ran Blekhman,et al.  Tracking the popularity and outcomes of all bioRxiv preprints , 2019, bioRxiv.

[14]  J. Bohannon.  Who's afraid of peer review? , 2013, Science.

[15]  S. Goodman,et al.  Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine , 1994, Annals of Internal Medicine.

[16]  Nader Shaikh,et al.  A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review , 2017, PloS one.

[17]  S. Wich,et al.  Great apes: Fresh strategies to save orangutans , 2016, Nature.

[18]  Athina Tatsioni,et al.  Who is afraid of reviewers’ comments? Or, why anything can be published and anything can be cited , 2010, European journal of clinical investigation.

[19]  D. Moher,et al.  Journal editors’ perspectives on the roles and tasks of peer reviewers in biomedical journals: a qualitative study , 2019, BMJ Open.

[20]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials , 2001, The Lancet.

[21]  Erik Cobo,et al.  Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial , 2007, PloS one.

[22]  D. Altman,et al.  Are Peer Reviewers Encouraged to Use Reporting Guidelines? A Survey of 116 Health Research Journals , 2012, PloS one.

[23]  David Moher,et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies , 2015, BMJ : British Medical Journal.

[24]  P. Haas,et al.  Sustainable Development Goals: create a coordinating body , 2016, Nature.

[25]  Alex Csiszar,et al.  Peer review: Troubled from the start , 2016, Nature.

[26]  Matthias Egger,et al.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies , 2007, PLoS medicine.

[27]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[28]  Malcolm R. Macleod.  Findings of a retrospective, controlled cohort study of the impact of a change in Nature journals' editorial policy for life sciences research on the completeness of reporting study design and execution , 2017 .

[29]  J. McKenzie,et al.  Study Quality guide , 2018 .

[30]  I. Cuthill,et al.  Reporting : The ARRIVE Guidelines for Reporting Animal Research , 2010 .

[31]  Richard Walker,et al.  Emerging trends in peer review—a survey , 2015, Front. Neurosci..

[32]  E. Tabor.  Prepublication culture in clinical research , 2016, The Lancet.

[33]  Paul Ginsparg,et al.  It was twenty years ago today , 2011, ArXiv.

[34]  Jing Liao,et al.  A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus) , 2019, Research Integrity and Peer Review.

[35]  Scott Chamberlain,et al.  Client for Various 'CrossRef' 'APIs' , 2016 .

[36]  Martin Klein,et al.  Comparing published scientific journal articles to their pre-print versions , 2016, 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL).

[37]  Ronald D. Vale,et al.  Accelerating scientific publication in biology , 2015, Proceedings of the National Academy of Sciences.

[38]  Philip E. Bourne,et al.  Preprints for the life sciences , 2016, Science.

[39]  M. Mahoney.  Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[40]  K. Black,et al.  bioRxiv: the preprint server for biology , 2019, bioRxiv.

[41]  Jing Liao,et al.  Did a change in Nature journals’ editorial policy for life sciences research improve reporting? , 2019, BMJ Open Science.

[42]  M. Cobb.  The prehistory of biology preprints: A forgotten experiment from the 1960s , 2017, PLoS biology.

[43]  Guy Beauchamp,et al.  ARRIVE has not ARRIVEd: Support for the ARRIVE (Animal Research: Reporting of in vivo Experiments) guidelines does not improve the reporting quality of papers in animal welfare, analgesia or anesthesia , 2018, PloS one.

[44]  E. Topol,et al.  Time for a prepublication culture in clinical research? , 2015, The Lancet.

[45]  Mitchell J. Nathan,et al.  Low agreement among reviewers evaluating the same NIH grant applications , 2018, Proceedings of the National Academy of Sciences.