Imperfect referees: Reducing the impact of multiple biases in peer review

Bias in peer review entails systematic prejudice that prevents accurate and objective assessment of scientific studies. The disparity between referees' opinions on the same paper typically makes it difficult to judge the paper's quality. This article presents a comprehensive study of peer review biases with regard to 2 aspects of referees: the static profiles (factual authority and self‐reported confidence) and the dynamic behavioral context (the temporal ordering of reviews by a single reviewer), exploiting anonymized, real‐world review reports of 2 different international conferences in information systems / computer science. Our work extends conventional bias research by considering multiple biases occurring simultaneously. Our findings show that the referees' static profiles are more dominant in peer review bias when compared to their dynamic behavioral context. Of the static profiles, self‐reported confidence improved both conference fitness and impact‐based bias reductions, while factual authority could only contribute to conference fitness‐based bias reduction. Our results also clearly show that the reliability of referees' judgments varies along their static profiles and is contingent on the temporal interval between 2 consecutive reviews.

[1]  A. Tversky,et al.  Prospect theory: an analysis of decision under risk — Source link , 2007 .

[2]  A. Furnham,et al.  A literature review of the anchoring effect , 2011 .

[3]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[4]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[5]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[6]  Jason P. Mitchell,et al.  The Seven Sins of Memory , 2003, Annals of the New York Academy of Sciences.

[7]  Norman Miller,et al.  Ten years of research on the false-consensus effect: an empirical and theoretical review , 1987 .

[8]  Hun-Tong Tan,et al.  Does Time Constraint Lead to Poorer Audit Performance? Effects of Forewarning of Impending Time Constraints and Instructions , 2011 .

[9]  D. Benos,et al.  The ups and downs of peer review. , 2007, Advances in physiology education.

[10]  P. Silvia.  Exploring the Psychology of Interest , 2006 .

[11]  Eric Medvet,et al.  Citation Counts and Evaluation of Researchers in the Internet Age , 2013, ArXiv.

[12]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[13]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[14]  Paul W. Paese,et al.  Peer Input and Revised Judgment: Exploring the Effects of (Un) Biased Confidence1 , 1993 .

[15]  Vincent Larivière,et al.  The weakening relationship between the impact factor and papers' citations in the digital age , 2012, J. Assoc. Inf. Sci. Technol..

[16]  Hermann Ebbinghaus,et al.  Memory: a contribution to experimental psychology. , 1987, Annals of neurosciences.

[17]  M. Bar-Hillel.  The base-rate fallacy in probability judgments. , 1980 .

[18]  Hermann Ebbinghaus (1885).  Memory: A Contribution to Experimental Psychology , 2013, Annals of Neurosciences.

[19]  Ulf Sandström,et al.  Cognitive Bias in Peer Review : a New Approach , 2009 .

[20]  M. Mahoney.  Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[21]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[22]  D. Schacter.  The seven sins of memory. Insights from psychology and cognitive neuroscience. , 1999, The American psychologist.

[23]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[24]  M Nylenna,et al.  Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. , 1994, JAMA.

[25]  Harold Maurice Collins,et al.  New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System , 1991 .

[26]  W. Stehbens,et al.  Basic philosophy and concepts underlying scientific peer review. , 1999, Medical hypotheses.

[27]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[28]  Joseph A. Konstan,et al.  Conference paper selectivity and impact , 2010, Commun. ACM.

[29]  Lutz Bornmann,et al.  A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? , 2010, Scientometrics.

[30]  R. Pohl.  Cognitive illusions : a handbook on fallacies and biases in thinking, judgement and memory , 2004 .

[31]  M. Hojat,et al.  Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process , 2003, Advances in health sciences education : theory and practice.

[32]  Liv Langfeldt,et al.  The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome , 2001, Peer review in an Era of Evaluation.

[33]  J. Baron.  Thinking and Deciding , 2023 .

[34]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[35]  Param Vir Singh,et al.  A Hidden Markov Model for Collaborative Filtering , 2010, MIS Q..

[36]  C. Lee Jones,et al.  Scientific Journals in the United States: Their Production, Use, and Economics , 1983 .

[37]  Yehuda Koren,et al.  Factorization meets the neighborhood: a multifaceted collaborative filtering model , 2008, KDD.

[38]  Eyal Peer,et al.  Can Multiple Biases Occur in a Single Situation? Evidence From Media Bias Research , 2012 .

[39]  Yehuda Koren,et al.  Collaborative filtering with temporal dynamics , 2009, KDD.

[40]  Mark Ware,et al.  Peer review in scholarly journals: Perspective of the scholarly community - Results from an international study , 2008, Inf. Serv. Use.

[41]  C. Ngamen-Kouassi,et al.  Introduction to Logic and Critical Thinking , 2010 .