Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study

Objective To investigate the effectiveness of open peer review as a mechanism to improve the reporting of randomised trials published in biomedical journals. Design Retrospective before and after study. Setting BioMed Central series medical journals. Sample 93 primary reports of randomised trials published in BMC-series medical journals in 2012. Main outcome measures Changes to the reporting of methodological aspects of randomised trials in manuscripts after peer review, based on the CONSORT checklist, corresponding peer reviewer reports, the type of changes requested, and the extent to which authors adhered to these requests. Results Of the 93 trial reports, 38% (n=35) did not describe the method of random sequence generation, 54% (n=50) concealment of allocation sequence, 50% (n=46) whether the study was blinded, 34% (n=32) the sample size calculation, 35% (n=33) specification of primary and secondary outcomes, 55% (n=51) results for the primary outcome, and 90% (n=84) details of the trial protocol. The number of changes between manuscript versions was relatively small; most involved adding new information or altering existing information. Most changes requested by peer reviewers had a positive impact on the reporting of the final manuscript—for example, adding or clarifying randomisation and blinding (n=27), sample size (n=15), primary and secondary outcomes (n=16), results for primary or secondary outcomes (n=14), and toning down conclusions to reflect the results (n=27). Some changes requested by peer reviewers, however, had a negative impact, such as adding additional unplanned analyses (n=15). Conclusion Peer reviewers fail to detect important deficiencies in reporting of the methods and results of randomised trials. The number of these changes requested by peer reviewers was relatively small. Although most had a positive impact, some were inappropriate and could have a negative impact on reporting in the final publication.

[1]  J. PérezMartín,et al.  [International Committee of Medical Journal Editors]. , 2008, Revista alergia Mexico.

[2]  Jane Friedman.  Suspended judgment , 2013 .

[3]  J Croat Med,et al.  Uniform requirements for manuscripts submitted to biomedical journals , 1988, Journal of manipulative and physiological therapeutics.

[4]  T. Jefferson,et al.  Editorial peer review for improving the quality of reports of biomedical studies. , 2007, The Cochrane database of systematic reviews.

[5]  David Moher,et al.  Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane reviewa , 2012, Systematic Reviews.

[6]  Suspended judgment. Editorial peer review: let us put it on trial. , 1992, Controlled clinical trials.

[7]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[8]  J. Carpenter,et al.  Effects of training on quality of peer review: randomised controlled trial , 2004, BMJ : British Medical Journal.

[9]  Sara Schroter,et al.  What errors do peer reviewers detect, and does training improve their ability to detect them? , 2008, Journal of the Royal Society of Medicine.

[10]  D. Altman,et al.  Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial , 2011, BMJ : British Medical Journal.

[11]  Douglas G Altman,et al.  Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. , 2010, JAMA.

[12]  Gordon H Guyatt,et al.  Credibility of claims of subgroup effects in randomised controlled trials: systematic review , 2012, BMJ : British Medical Journal.

[13]  Erik Cobo,et al.  Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial , 2007, PloS one.

[14]  D. Rennie,et al.  Masking author identity in peer review: what factors influence masking success? PEER Investigators. , 1998, JAMA.

[15]  R Smith,et al.  Opening up BMJ peer review , 1999, BMJ.

[16]  Kay Dickersin,et al.  What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion , 2007, BMC medical research methodology.

[17]  D. Ratner,et al.  Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi‐rater study , 2011, The British journal of dermatology.

[18]  Sun Huh,et al.  Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice , 2008, Journal of Educational Evaluation for Health Professions.

[19]  Sally Hopewell,et al.  The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed , 2010, BMJ : British Medical Journal.

[20]  A. Urrutia,et al.  Efecto de la revisión estadística en la calidad de los manuscritos publicados en Medicina Clínica: estudio aleatorizado , 2003 .

[21]  William M. Tierney,et al.  Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care? , 2010, PloS one.

[22]  N. Black,et al.  What makes a good reviewer and a good review for a general medical journal? , 1998, JAMA.