Reviewing the reviewers: A study of author perception on peer reviews in computer science

Peer reviewing is an important form of collaborative work that is used for quality assurance in science and in other domains like software development and knowledge management. Review ratings by authors have potential to improve the quality of peer reviews, by giving way to remuneration of good reviews. A significant problem, however, is that authors' perception is hardly neutral, but might be affected by the reviews. To gain insight into their perception of peer reviews, we have conducted a survey among the authors of papers submitted to a peer-reviewed computer science conference. One of our findings is that authors are satisfied with reviews whose comments they deem helpful, and when they feel that the reviewer has made an effort to understand the paper. Suprisingly, these results hold when controlled for the score given by the reviewer. Based on the study results, we discuss the suitability of author ratings to identify high-quality reviews. We describe a remuneration function for reviews based on author ratings that aims to neutralize the effects of review scores.

[1]  Klemens Böhm,et al.  Peer production of structured knowledge -: an empirical study of ratings and incentive mechanisms , 2008, CIKM '08.

[2]  Sara Ellis Simonsen,et al.  Author Perception of Peer Review , 2008, Obstetrics and gynecology.

[3]  R. Dawes.  Statistical criteria for establishing a truly false consensus effect , 1989 .

[4]  E. Lawson,et al.  Effect of acceptance or rejection on the author's evaluation of peer review of medical manuscripts. , 1990, JAMA.

[5]  Ramanathan V. Guha,et al.  User Ratings of Ontologies: Who Will Rate the Raters? , 2005, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[6]  Timo Hannay,et al.  Nature's Peer Review Debate , 2006 .

[7]  J F Waeckerle,et al.  Reliability of editors' subjective quality ratings of peer reviews of manuscripts. , 1998, JAMA.

[8]  Karl E. Wiegers,et al.  Peer Reviews in Software: A Practical Guide , 2001 .

[9]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[10]  Andy Huber,et al.  Peer reviews in software: a practical guide , 2002, SOEN.

[11]  M. Callaham,et al.  Author perception of peer review: impact of review quality and acceptance on satisfaction. , 2002, JAMA.

[12]  Michael Derntl,et al.  The student view on online peer reviews , 2009, ITiCSE.

[13]  D. Prelec.  A Bayesian Truth Serum for Subjective Data , 2004, Science.

[14]  John Riedl,et al.  Is seeing believing?: how recommender system interfaces affect users' opinions , 2003, CHI '03.

[15]  Michael L Callaham,et al.  Effect of written feedback by editors on quality of reviews: two randomized trials. , 2002, JAMA.

[16]  Norman Miller,et al.  Ten years of research on the false-consensus effect: an empirical and theoretical review , 1987 .

[17]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[18]  Jerome L. Myers,et al.  Research Design and Statistical Analysis , 1991 .

[19]  Daniel Galin,et al.  Software Quality Assurance: From Theory to Implementation , 2003 .