Bias in peer review

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to contend with the complexity of social interactions among actors involved directly and indirectly in peer review. © 2013 Wiley Periodicals, Inc.

[1]  J. Ioannidis,et al.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[2]  Cassidy R. Sugimoto,et al.  Citation gamesmanship: testing for evidence of ego bias in peer review , 2013, Scientometrics.

[3]  Carole J. Lee A Kuhnian Critique of Psychometric Research on Peer Review , 2012 .

[4]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[5]  C. Sen Rebound peer review: a viable recourse for aggrieved authors? , 2012, Antioxidants & redox signaling.

[6]  Chris I. Baker,et al.  Toward a New Model of Scientific Publishing: Discussion and a Proposal , 2011, Front. Comput. Neurosci..

[7]  Lutz Bornmann,et al.  A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication? , 2011 .

[8]  J. Havey,et al.  Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi‐rater study , 2011, The British journal of dermatology.

[9]  R. Frodeman,et al.  Peer review and the ex ante assessment of societal impacts , 2011 .

[10]  B. Lawrence,et al.  Citation and Peer Review of Data: Moving Towards Formal Data Publication , 2011, Int. J. Digit. Curation.

[11]  Kathlyn E. Fletcher,et al.  The Validity of Peer Review in a General Medicine Journal , 2011, PloS one.

[12]  Laura Valkonen,et al.  Gender balance in Cortex acceptance rates , 2011, Cortex.

[13]  T. Rees,et al.  The Gendered Construction of Scientific Excellence , 2011 .

[14]  Christian D. Schunn,et al.  Social Biases and Solutions for Procedural Objectivity , 2011, Hypatia.

[15]  S. Ceci,et al.  Understanding current causes of women's underrepresentation in science , 2011, Proceedings of the National Academy of Sciences.

[16]  Allan Kardec Duailibe,et al.  No crisis in supply of peer reviewers , 2010, Nature.

[17]  L. Bornmann,et al.  A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants , 2010, PloS one.

[18]  Axel Boldt,et al.  Extending ArXiv.org to Achieve Open Peer Review and Publishing , 2010, ArXiv.

[19]  S. Evans,et al.  Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial , 2010, BMJ : British Medical Journal.

[20]  Lutz Bornmann,et al.  Do Author-Suggested Reviewers Rate Submissions More Favorably than Editor-Suggested Reviewers? A Study on Atmospheric Chemistry and Physics , 2010, PloS one.

[21]  Rebecca S. Benner,et al.  Differences in editorial board reviewer behavior based on gender. , 2010, Journal of women's health.

[22]  Stefan Thurner,et al.  Peer-review in a world with rational scientists: Toward selection of the average , 2010, 1008.4324.

[23]  Ruth E. Duerr,et al.  Data Citation and Peer Review , 2010 .

[24]  David M. Schultz,et al.  Are three heads better than two? How the number of reviewers and editor behavior affect the rejection rate , 2010, Scientometrics.

[25]  Lutz Bornmann,et al.  A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? , 2010, Scientometrics.

[26]  D. Fanelli Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data , 2010, PloS one.

[27]  William M. Tierney,et al.  Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care? , 2010, PloS one.

[28]  Lutz Bornmann,et al.  Reliability of reviewers' ratings when using public peer review: a case study , 2010, Learn. Publ..

[29]  David M. Schultz,et al.  REjEctIOn RAtES FOR jOURnAlS PUBlISHInG In tHE AtMOSPHERIc ScIEncES , 2010 .

[30]  Stefan Hirschauer,et al.  Editorial Judgments , 2010 .

[31]  Amber E. Budden,et al.  To Name or Not to Name: The Effect of Changing Author Gender on Peer Review , 2009 .

[32]  Lutz Bornmann,et al.  Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition , 2009, J. Informetrics.

[33]  L. Bornmann,et al.  Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches , 2009 .

[34]  Fabio Casati,et al.  Is peer review any good? A quantitative analysis of peer review , 2009 .

[35]  D. Linden,et al.  Is There Gender Bias in the Peer Review Process at Journal of Neurophysiology , 2009 .

[36]  Lutz Bornmann,et al.  The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models , 2009, Scientometrics.

[37]  Sergio Sismondo,et al.  Ghosts in the Machine , 2009, Social studies of science.

[38]  Yoshiyuki Takeda,et al.  Comparative study on methods of detecting research fronts using different types of citation , 2009 .

[39]  S. Isenberg,et al.  The effect of masking manuscripts for the peer-review process of an ophthalmic journal , 2009, British Journal of Ophthalmology.

[40]  M. Lamont,et al.  Fairness as Appropriateness , 2009 .

[41]  T. Ley,et al.  The Gender Gap in NIH Grant Applications , 2008, Science.

[42]  Molly C Dougherty,et al.  Blinding in peer review: the preferences of reviewers for nursing journals. , 2008, Journal of advanced nursing.

[43]  Lutz Bornmann,et al.  How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study , 2008, J. Informetrics.

[44]  L. Bornmann,et al.  The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. , 2008, Angewandte Chemie.

[45]  R. Whittaker,et al.  Journal review and gender equality: a critical comment on Budden et al. , 2008, Trends in ecology & evolution.

[46]  Leif Engqvist,et al.  Double-blind peer review and gender publication bias , 2008, Animal Behaviour.

[47]  Lokman I. Meho,et al.  A proposal for a dynamic h-type index , 2008 .

[48]  Alan S. Gerber,et al.  Publication Bias in Empirical Sociological Research , 2008 .

[49]  Robert P Freckleton,et al.  Does double-blind review benefit female authors? , 2008, Trends in ecology & evolution.

[50]  U. W. Jayasinghe,et al.  Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. , 2008, The American psychologist.

[51]  Brian Ahl,et al.  Sociological Reflections on My Work Experience , 2008 .

[52]  David Lane Double-blind review: easy to guess in specialist fields , 2008, Nature.

[53]  Ulf Sandström,et al.  Persistent nepotism in peer-review , 2008, Scientometrics.

[54]  Richard J. C. Brown,et al.  Double anonymity in peer review within the chemistry periodicals community , 2007, Learn. Publ..

[55]  Irene Hames,et al.  Peer Review and Manuscript Management in Scientific Journals , 2007 .

[56]  Leo Egghe,et al.  Uncertainty and information: Foundations of generalized information theory: Book Reviews , 2007 .

[57]  L. Bornmann,et al.  Gender differences in grant peer review: A meta-analysis , 2007, J. Informetrics.

[58]  Lutz Bornmann,et al.  Potential sources of bias in research fellowship assessments: effects of university prestige and field of study , 2006 .

[59]  M. Kon,et al.  Who benefits from peer review? An analysis of the outcome of 100 requests for review by Plastic and Reconstructive Surgery. , 2006, Plastic and reconstructive surgery.

[60]  M. Hakel,et al.  An Examination of Sources of Peer-Review Bias , 2006, Psychological science.

[61]  C. Gross,et al.  Effect of blinded peer review on abstract acceptance. , 2006, JAMA.

[62]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[63]  J. Bargh,et al.  The Automaticity of Social Life , 2006, Current directions in psychological science.

[64]  Hans-Dieter Daniel,et al.  Publications as a measure of scientific advancement and of scientists' productivity , 2005, Learn. Publ..

[65]  D. Shatz Peer Review: A Critical Inquiry , 2004 .

[66]  H. Lacey Is Science Value Free?: Values and Scientific Understanding , 2004 .

[67]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[68]  B. Cronin Scholarly communication and epistemic cultures , 2003 .

[69]  Michael Mabe,et al.  The growth and number of journals , 2003 .

[70]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[71]  Harald Merckelbach,et al.  Peer-Review: Let's Imitate the Lawyers! , 2002, Cortex.

[72]  Joseph Smith,et al.  Impact of blinded versus unblinded abstract review on scientific program content. , 2002, The Journal of urology.

[73]  Tom Tregenza,et al.  Gender bias in the refereeing process , 2002 .

[74]  R. Spier The history of the peer-review process. , 2002, Trends in biotechnology.

[75]  D. Rennie,et al.  Publication bias in editorial decision making. , 2002, JAMA.

[76]  Tony Delamothe,et al.  Twenty thousand conversations , 2002, BMJ : British Medical Journal.

[77]  M. Biagioli From Book Censorship to Academic Peer Review , 2002 .

[78]  R. Melero,et al.  Referees’ Attitudes toward Open Peer Review and Electronic Transmission of Papers , 2001 .

[79]  H. Longino The Fate of Knowledge , 2001 .

[80]  Dale T. Miller,et al.  Moral credentials and the expression of prejudice. , 2001, Journal of personality and social psychology.

[81]  A. F. J. Van Raan,et al.  Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research , 2001 .

[82]  A. Palmer,et al.  QUASIREPLICATION AND THE CONTRACT OF ERROR: Lessons from Sex Ratios, Heritabilities and Fluctuating Asymmetry , 2000 .

[83]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[84]  J. Ziman Real Science: What It Is and What It Means , 2000 .

[85]  E Ernst,et al.  A randomized controlled study of reviewer bias against an unconventional therapy , 2000, Journal of the Royal Society of Medicine.

[86]  G. Wilkinson,et al.  Open peer review: A randomised controlled trial , 2000, British Journal of Psychiatry.

[87]  Fiona Godlee,et al.  Effect of blinding and unmasking on the quality of peer review , 1999, Journal of General Internal Medicine.

[88]  R. Steinpreis,et al.  The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study , 1999 .

[89]  E. Ernst,et al.  Reviewer bias against the unconventional? A randomized double-blind study of peer review. , 1999, Complementary therapies in medicine.

[90]  Antonio J. Herrera Language bias discredits the peer-review system , 1999, Nature.

[91]  Richard N. Zare,et al.  Interdisciplinary Research: From Belief to Reality , 1999, Science.

[92]  R Smith,et al.  Opening up BMJ peer review , 1999, BMJ.

[93]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[94]  Yu Xie,et al.  Sex differences in research productivity : New evidence about an old puzzle , 1998 .

[95]  A. Bardy Bias in reporting clinical trials. , 1998, British journal of clinical pharmacology.

[96]  Simon Wessely,et al.  Peer review of grant applications: what do we know? , 1998, The Lancet.

[97]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[98]  D. Rennie,et al.  Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. , 1998, JAMA.

[99]  A. Link US and non-US submissions: an analysis of reviewer bias. , 1998, JAMA.

[100]  N. Black,et al.  Effect of blinding and unmasking on the quality of peer review: a randomized trial. , 1998, JAMA.

[101]  R. Nickerson Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[102]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 2 , 1998 .

[103]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 1 , 1998 .

[104]  J. Ioannidis Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. , 1998, JAMA.

[105]  H. Friesen Equal opportunities in Canada , 1998, Nature.

[106]  M. Biagioli,et al.  LIFE SCIENCES FORUM The Instability of Authorship: Credit and Responsibility in Contemporary Biomedicine , 1998 .

[107]  J. Grant,et al.  No evidence of sexism in peer review , 1997, Nature.

[108]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[109]  E Frank,et al.  Editors' requests of peer reviewers: a study and a proposal. , 1996, Preventive medicine.

[110]  S. Shapin,et al.  A Social History of Truth: Civility and Science in Seventeenth-Century England, Steven Shapin. 1994. University of Chicago, Chicago, IL. 530 pages. ISBN: 0-226-75018-3. $29.95 , 1995 .

[111]  Juan Miguel Campanario,et al.  Commentary: On Influential Books and Journal Articles Initially Rejected Because of Negative Referees' Evaluations , 1995 .

[112]  J. R. Gilbert,et al.  Is there gender bias in JAMA's peer review process? , 1994, JAMA.

[113]  S. Goodman,et al.  Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine , 1994, Annals of Internal Medicine.

[114]  P. Kitcher,et al.  The Advancement of Science: Science Without Legend, Objectivity without Illusion. , 1994 .

[115]  Daniel S. Hamermesh,et al.  Facts and Myths about Refereeing , 1994 .

[116]  David N. Laband,et al.  Favoritism versus Search for Good Papers: Empirical Evidence Regarding the Behavior of Journal Editors , 1994, Journal of Political Economy.

[117]  Blaise Cronin,et al.  The trajectory of rejection , 1992, J. Documentation.

[118]  R. Liang,et al.  Recovery from Chronic Hepatitis B , 1992, Annals of Internal Medicine.

[119]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[120]  MARGOT O'TOOLE,et al.  Imanishi-Kari (continued) , 1991, Nature.

[121]  Harold Maurice Collins,et al.  New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System , 1991 .

[122]  A. Yankauer,et al.  How blind is blind review? , 1991, American journal of public health.

[123]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[124]  John C. Bailar,et al.  Reliability, fairness, objectivity and other inappropriate goals in peer review , 1991, Behavioral and Brain Sciences.

[125]  Lowell L. Hargens,et al.  Neglected considerations in the analysis of agreement among journal referees , 1990, Scientometrics.

[126]  D F Horrobin,et al.  The philosophical basis of peer review and the suppression of innovation. , 1990, JAMA.

[127]  K. Dickersin The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[128]  D. Kronick Peer review in 18th-century scientific journalism. , 1990, JAMA.

[129]  M J Gardner,et al.  An exploratory study of statistical assessment of papers published in the British Medical Journal. , 1990, JAMA.

[130]  David L. Hull,et al.  Science as a Process: An Evolutionary Account of the Social and Conceptual Development of Science, David L. Hull. 1988. The University of Chicago Press, Chicago, IL. 608 pages. ISBN: 0-226-35060-4. $39.95 , 1989 .

[131]  J. McCullough,et al.  First Comprehensive Survey of NSF Applicants Focuses on Their Concerns About Proposal Review , 1989 .

[132]  A. Greenwald,et al.  Under what conditions does theory obstruct research progress? , 1986, Psychological review.

[133]  M. Ferber,et al.  Citations: Are They an Objective Measure of Scholarly Merit? , 1986, Signs: Journal of Women in Culture and Society.

[134]  Alan L. Porter,et al.  Peer Review of Interdisciplinary Research Proposals , 1985 .

[135]  Daryl E. Chubin,et al.  Experience with NIH Peer Review: Researchers' Cynicism and Desire for Change , 1985 .

[136]  Stephen J. Ceci,et al.  How blind is blind review , 1984 .

[137]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[138]  Stevan Harnad,et al.  Peer Commentary on Peer Review: A Case Study in Scientific Quality Control , 1983 .

[139]  Stephen J. Ceci,et al.  Peer-review research: Objections and obligations , 1982, Behavioral and Brain Sciences.

[140]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[141]  J. R. Cole,et al.  Chance and consensus in peer review. , 1981, Science.

[142]  Robert F. Rich,et al.  Who Is Making Science Policy , 1979 .

[143]  L J Carter,et al.  A new and searching look at NSF. , 1979, Science.

[144]  Stephen D. Gottfredson,et al.  Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. , 1978 .

[145]  M. Mahoney Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[146]  Derek de Solla Price,et al.  A general theory of bibliometric and other cumulative advantage processes , 1976, J. Am. Soc. Inf. Sci..

[147]  T. Gustafson The Controversy Over Peer Review , 1975, Science.

[148]  J Walsh,et al.  NSF and Its Critics in Congress: New Pressure on Peer Review. , 1975, Science.

[149]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1973 .

[150]  W. Hagstrom Inputs, Outputs, and the Prestige of University Science Departments , 1971 .

[151]  R. Merton The Matthew Effect in Science , 1968, Science.

[152]  Nigel W. Bond,et al.  Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model , 2011, J. Informetrics.

[153]  Lutz Bornmann,et al.  Peer Review and Bibliometric: Potentials and Problems , 2011 .

[154]  Bo-Christer Björk,et al.  Scientific journal publishing: yearly volume and open access availability , 2009, Inf. Res..

[155]  Ulf Sandström,et al.  Cognitive Bias in Peer Review : a New Approach , 2009 .

[156]  Michèle Lamont,et al.  How Professors Think: Inside the Curious World of Academic Judgment , 2009 .

[157]  T. Tregenza,et al.  Double-blind review favours increased representation of female authors. , 2008, Trends in ecology & evolution.

[158]  Lutz Bornmann,et al.  Gatekeepers of science - Effects of external reviewers' attributes on the assessments of fellowship applications , 2007, J. Informetrics.

[159]  T. Tyler,et al.  Psychological perspectives on legitimacy and legitimation. , 2006, Annual review of psychology.

[160]  Julie Thompson Klein,et al.  Promoting Interdisciplinary Research The Case of the Academy of Finland , 2005 .

[161]  Blaise Cronin,et al.  Hyperauthorship: A postmodern perversion or evidence of a structural shift in scholarly communication practices? , 2001, J. Assoc. Inf. Sci. Technol..

[162]  S. Chaiken,et al.  Dual-process theories in social psychology , 1999 .

[163]  Joshua S. Gans,et al.  Why Referees Are Not Paid (Enough) , 1998 .

[164]  Hans-Dieter Daniel,et al.  Guardians of Science: Fairness and Reliability of Peer Review , 1994 .

[165]  R. Blank The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[166]  Samuel Ball,et al.  The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Interjudgmental Reliability , 1989 .