On a Reliable Peer-Review Process

We propose an enhanced peer-review process where the reviewers are encouraged to truthfully disclose their reviews. We start by modelling that process using a Bayesian model where the uncertainty regarding the quality of the manuscript is taken into account. After that, we introduce a scoring function to evaluate the reported reviews. Under mild assumptions, we show that reviewers strictly maximize their expected scores by telling the truth. We also show how those scores can be used in order to reach consensus.

[1]  Richards J. Heuer,et al.  Psychology of Intelligence Analysis , 1999 .

[2]  Yoav Shoham,et al.  Multiagent Systems - Algorithmic, Game-Theoretic, and Logical Foundations , 2009 .

[3]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[4]  Ward Edwards,et al.  Judgment under uncertainty: Conservatism in human information processing , 1982 .

[5]  Jörg Rothe,et al.  How to Calibrate the Scores of Biased Reviewers by Quadratic Programming , 2011, AAAI.

[6]  John R. Douceur.  Paper rating vs. paper ranking , 2008, OPSR.

[7]  Paul Resnick,et al.  Eliciting Informative Feedback: The Peer-Prediction Method , 2005, Manag. Sci..

[8]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[9]  Matthew E Falagas,et al.  How masked is the "masked peer review" of abstracts submitted to international medical conferences? , 2006, Mayo Clinic proceedings.

[10]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[11]  M. Degroot.  Reaching a Consensus , 1974 .

[12]  Richard E. Abel,et al.  Scholarly Publishing: Books, Journals, Publishers, and Libraries in the Twentieth Century , 2001 .

[13]  M. Bouton,et al.  Masked Reviews Are Not Fairer Reviews , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[14]  R. Zeckhauser,et al.  Efficiency Despite Mutually Payoff-Relevant Private Information: The Finite Case , 1990 .

[15]  A. Weller.  Editorial peer review : its strengths and weaknesses , 2001 .

[16]  D. Rennie,et al.  Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. , 1998, JAMA.

[17]  E. Eisenberg,et al.  CONSENSUS OF SUBJECTIVE PROBABILITIES: THE PARI-MUTUEL METHOD, , 1959 .

[18]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[19]  R. Primack,et al.  Bias in the review process , 2008 .

[20]  A. Yankauer,et al.  How blind is blind review? , 1991, American journal of public health.

[21]  Gregory B. Newby,et al.  Scholarly publishing: the electronic frontier , 1996 .

[22]  Torsten Norvig.  Consensus of Subjective Probabilities: A Convergence Theorem , 1967 .

[23]  N. Black,et al.  Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. , 1999, Journal of clinical epidemiology.

[24]  Ariel Rubinstein,et al.  A Course in Game Theory , 1995 .

[25]  R. L. Winkler.  Scoring Rules and the Evaluation of Probability Assessors , 1969 .