Bias in peer review

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to contend with the complexity of social interactions among actors involved directly and indirectly in peer review. © 2013 Wiley Periodicals, Inc.

[1]  Ulrich Teichler,et al.  University rankings : theoretical basis, methodology and impacts on global higher education , 2011 .

[2]  A. F. J. Van Raan,et al.  Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research , 2001 .

[3]  R. Frodeman,et al.  Peer review and the ex ante assessment of societal impacts , 2011 .

[4]  Lowell L. Hargens,et al.  Neglected considerations in the analysis of agreement among journal referees , 1990, Scientometrics.

[5]  D. Fanelli Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data , 2010, PloS one.

[6]  Axel Boldt,et al.  Extending ArXiv.org to Achieve Open Peer Review and Publishing , 2010, ArXiv.

[7]  S. Chaiken,et al.  Dual-process theories in social psychology , 1999 .

[8]  Hans-Dieter Daniel,et al.  Publications as a measure of scientific advancement and of scientists' productivity , 2005, Learn. Publ..

[9]  D. Linden,et al.  Is There Gender Bias in the Peer Review Process at Journal of Neurophysiology , 2009 .

[10]  R. Spier The history of the peer-review process. , 2002, Trends in biotechnology.

[11]  R. Merton The Matthew Effect in Science , 1968, Science.

[12]  M. Hagberg Editorial , 2004 .

[13]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[14]  J. Grant,et al.  No evidence of sexism in peer review , 1997, Nature.

[15]  Yu Xie,et al.  Sex differences in research productivity : New evidence about an old puzzle , 1998 .

[16]  Philip Kitcher,et al.  Precis of The Advancement of Science@@@The Advancement of Science: Science without Legend, Objectivity without Illusions. , 1995 .

[17]  A. Oswald,et al.  Can We Test for Bias in Scientific Peer-Review? , 2008 .

[18]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[19]  R. Melero,et al.  Referees’ Attitudes toward Open Peer Review and Electronic Transmission of Papers , 2001 .

[20]  B. Cronin Scholarly communication and epistemic cultures , 2003 .

[21]  L. Bornmann,et al.  The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. , 2008, Angewandte Chemie.

[22]  Stacy M Carter,et al.  Journal peer review in context: A qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing. , 2011, Social science & medicine.

[23]  L. Bornmann,et al.  A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants , 2010, PloS one.

[24]  C. Gross,et al.  Effect of blinded peer review on abstract acceptance. , 2006, JAMA.

[25]  Lutz Bornmann,et al.  Gatekeepers of science - Effects of external reviewers' attributes on the assessments of fellowship applications , 2007, J. Informetrics.

[26]  Juan Miguel Campanario,et al.  Commentary: On Influential Books and Journal Articles Initially Rejected Because of Negative Referees' Evaluations , 1995 .

[27]  Dale T. Miller,et al.  Moral credentials and the expression of prejudice. , 2001, Journal of personality and social psychology.

[28]  S. Fletcher Guardians of Science: Fairness and Reliability of Peer Review , 1994 .

[29]  A. Yankauer,et al.  How blind is blind review? , 1991, American journal of public health.

[30]  D. Rennie,et al.  Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. , 1998, JAMA.

[31]  Women in neuroscience: a numbers game , 2006, Nature Neuroscience.

[32]  Kathlyn E. Fletcher,et al.  The Validity of Peer Review in a General Medicine Journal , 2011, PloS one.

[33]  Irene Hames,et al.  Peer Review and Manuscript Management in Scientific Journals , 2007 .

[34]  Harald Merckelbach,et al.  Peer-Review: Let's Imitate the Lawyers! , 2002, Cortex.

[35]  M J Gardner,et al.  An exploratory study of statistical assessment of papers published in the British Medical Journal. , 1990, JAMA.

[36]  Rebecca S. Benner,et al.  Differences in editorial board reviewer behavior based on gender. , 2010, Journal of women's health.

[37]  H. Longino Science as Social Knowledge , 1990 .

[38]  Daryl E. Chubin,et al.  Experience with NIH Peer Review: Researchers' Cynicism and Desire for Change , 1985 .

[39]  Christian D. Schunn,et al.  Social Biases and Solutions for Procedural Objectivity , 2011, Hypatia.

[40]  Stefan Thurner,et al.  Peer-review in a world with rational scientists: Toward selection of the average , 2010, 1008.4324.

[41]  Stephen J. Ceci,et al.  Peer-review research: Objections and obligations , 1982, Behavioral and Brain Sciences.

[42]  Lutz Bornmann,et al.  A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? , 2010, Scientometrics.

[43]  Blaise Cronin,et al.  The trajectory of rejection , 1992, J. Documentation.

[44]  Lutz Bornmann,et al.  Gender differences in grant peer review: A meta-analysis , 2007, J. Informetrics.

[45]  L J Carter,et al.  A new and searching look at NSF. , 1979, Science.

[46]  K. Dickersin The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[47]  John C. Bailar,et al.  Reliability, fairness, objectivity and other inappropriate goals in peer review , 1991, Behavioral and Brain Sciences.

[48]  M. Ferber,et al.  Citations: Are They an Objective Measure of Scholarly Merit? , 1986, Signs: Journal of Women in Culture and Society.

[49]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[50]  M. Lamont,et al.  Fairness as Appropriateness , 2009 .

[51]  D F Horrobin,et al.  The philosophical basis of peer review and the suppression of innovation. , 1990, JAMA.

[52]  Alan S. Gerber,et al.  Publication Bias in Empirical Sociological Research , 2008 .

[53]  Brian Ahl,et al.  Sociological Reflections on My Work Experience , 2008 .

[54]  Robert P Freckleton,et al.  Does double-blind review benefit female authors? , 2008, Trends in ecology & evolution.

[55]  A. Palmer,et al.  QUASIREPLICATION AND THE CONTRACT OF ERROR: Lessons from Sex Ratios, Heritabilities and Fluctuating Asymmetry , 2000 .

[56]  Julie Thompson Klein,et al.  Promoting Interdisciplinary Research The Case of the Academy of Finland , 2005 .

[57]  W. Hagstrom Inputs, Outputs, and the Prestige of University Science Departments , 1971 .

[58]  Cassidy R. Sugimoto,et al.  Citation gamesmanship: testing for evidence of ego bias in peer review , 2012, Scientometrics.

[59]  Stephen J. Ceci,et al.  How blind is blind review , 1984 .

[60]  Lutz Bornmann,et al.  The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models , 2009, Scientometrics.

[61]  David M. Schultz,et al.  Are three heads better than two? How the number of reviewers and editor behavior affect the rejection rate , 2010, Scientometrics.

[62]  Joshua S. Gans,et al.  Why Referees Are Not Paid (Enough) , 1998 .

[63]  M. Hakel,et al.  An Examination of Sources of Peer-Review Bias , 2006, Psychological science.

[64]  Fabio Casati,et al.  Is peer review any good? A quantitative analysis of peer review , 2009 .

[65]  R. Blank The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[66]  H. Friesen Equal opportunities in Canada , 1998, Nature.

[67]  Leif Engqvist,et al.  Double-blind peer review and gender publication bias , 2008, Animal Behaviour.

[68]  William M. Tierney,et al.  Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care? , 2010, PloS one.

[69]  David L. Hull,et al.  Science as a Process: An Evolutionary Account of the Social and Conceptual Development of Science, David L. Hull. 1988. The University of Chicago Press, Chicago, IL. 608 pages. ISBN: 0-226-35060-4. $39.95 , 1989 .

[70]  T. Rees,et al.  The Gendered Construction of Scientific Excellence , 2011 .

[71]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[72]  Álvaro Santana Acuña How Professors Think: Inside the Curious World of Academic Judgment , 2012 .

[73]  A. Bardy Bias in reporting clinical trials. , 1998, British journal of clinical pharmacology.

[74]  Cedric J. Pearce Pioneering Research: A Risk Worth Taking By Donald W. Braben (Venture Research International Ltd. and University College, London). John Wiley & Sons Inc., Hoboken. 2004. x + 198 pp. 8 × 5.25 in. $39.95. ISBN 0-471-48852-6. , 2005 .

[75]  Ulf Sandström,et al.  Persistent nepotism in peer-review , 2008, Scientometrics.

[76]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006 .

[77]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[78]  Harold Maurice Collins,et al.  New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System , 1991 .

[79]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[80]  Antonio J. Herrera Language bias discredits the peer-review system , 1999, Nature.

[81]  Tony Delamothe,et al.  Twenty thousand conversations , 2002, BMJ : British Medical Journal.

[82]  Alan L. Porter,et al.  Peer Review of Interdisciplinary Research Proposals , 1985 .

[83]  Sergio Sismondo,et al.  Ghosts in the Machine , 2009, Social studies of science.

[84]  T. Tyler,et al.  Psychological perspectives on legitimacy and legitimation. , 2006, Annual review of psychology.

[85]  H. Longino The Fate of Knowledge , 2001 .

[86]  MARGOT O'TOOLE,et al.  Imanishi-Kari (continued) , 1991, Nature.

[87]  Lutz Bornmann,et al.  A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication? , 2011 .

[88]  Laura Valkonen,et al.  Gender balance in Cortex acceptance rates , 2011, Cortex.

[89]  H. Lacey Is Science Value Free?: Values and Scientific Understanding , 2004 .

[90]  Samuel Ball,et al.  The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Interjudgmental Reliability , 1989 .

[91]  Chris I. Baker,et al.  Toward a New Model of Scientific Publishing: Discussion and a Proposal , 2011, Front. Comput. Neurosci..

[92]  Norman Hackerman,et al.  Peerless Science: Peer Review and U.S. Science Policy , 1992 .

[93]  D. Kronick Peer review in 18th-century scientific journalism. , 1990, JAMA.

[94]  N. Black,et al.  Effect of blinding and unmasking on the quality of peer review: a randomized trial. , 1998, JAMA.

[95]  Lutz Bornmann,et al.  Reliability of reviewers' ratings when using public peer review: a case study , 2010, Learn. Publ..

[96]  H. Marsh,et al.  Improving the Peer-review Process for Grant Applications , 2022 .

[97]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[98]  S. Goodman,et al.  Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine , 1994, Annals of Internal Medicine.

[99]  Lutz Bornmann,et al.  How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study , 2008, J. Informetrics.

[100]  Lutz Bornmann,et al.  Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories , 2008 .

[101]  B. Ferguson,et al.  Modern psychometrics. The science of psychological assessment , 1990 .

[102]  Ronald E LaPorte,et al.  Peer review of grant applications , 1998, The Lancet.

[103]  Tom Tregenza,et al.  Gender bias in the refereeing process , 2002 .

[104]  S. Ceci,et al.  Understanding current causes of women's underrepresentation in science , 2011, Proceedings of the National Academy of Sciences.

[105]  Mark Ware,et al.  Peer review in scholarly journals: Perspective of the scholarly community - Results from an international study , 2008, Inf. Serv. Use.

[106]  Robert F. Rich,et al.  Who Is Making Science Policy , 1979 .

[107]  A. Greenwald,et al.  Under what conditions does theory obstruct research progress? , 1986, Psychological review.

[108]  Sarah Callaghan,et al.  Citation and Peer Review of Data: Moving Towards Formal Data Publication , 2011, Int. J. Digit. Curation.

[109]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[110]  D. Shatz Peer Review: A Critical Inquiry , 2004 .

[111]  J. Bargh,et al.  The Automaticity of Social Life , 2006, Current directions in psychological science.

[112]  R Smith,et al.  Opening up BMJ peer review , 1999, BMJ.

[113]  L. Bornmann,et al.  Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches , 2009 .

[114]  Carole J. Lee A Kuhnian Critique of Psychometric Research on Peer Review , 2012 .

[115]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations.@@@Einstein and the Generations of Science. , 1976 .

[116]  Amber E. Budden,et al.  To Name or Not to Name: The Effect of Changing Author Gender on Peer Review , 2009 .

[117]  Lutz Bornmann,et al.  Selecting manuscripts for a high-impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere , 2008, J. Assoc. Inf. Sci. Technol..

[118]  S. Isenberg,et al.  The effect of masking manuscripts for the peer-review process of an ophthalmic journal , 2009, British Journal of Ophthalmology.

[119]  D. Ratner,et al.  Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi‐rater study , 2011, The British journal of dermatology.

[120]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[121]  R. Nickerson Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[122]  G. Wilkinson,et al.  Open peer review: A randomised controlled trial , 2000, British Journal of Psychiatry.

[123]  J. McCullough,et al.  First Comprehensive Survey of NSF Applicants Focuses on Their Concerns About Proposal Review , 1989 .

[124]  Blaise Cronin,et al.  Hyperauthorship: A postmodern perversion or evidence of a structural shift in scholarly communication practices? , 2001, J. Assoc. Inf. Sci. Technol..

[125]  E. Ernst,et al.  Reviewer bias against the unconventional? A randomized double-blind study of peer review. , 1999, Complementary therapies in medicine.

[126]  T. Ley,et al.  The Gender Gap in NIH Grant Applications , 2008, Science.

[127]  J. R. Cole,et al.  Chance and consensus in peer review. , 1981, Science.

[128]  T. Tregenza,et al.  Double-blind review favours increased representation of female authors. , 2008, Trends in ecology & evolution.

[129]  Blaise Cronin,et al.  Vernacular and vehicular language , 2009, J. Assoc. Inf. Sci. Technol..

[130]  A. Link US and non-US submissions: an analysis of reviewer bias. , 1998, JAMA.

[131]  Tony Delamothe,et al.  Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial , 2010, BMJ : British Medical Journal.

[132]  Nigel W. Bond,et al.  Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model , 2011, J. Informetrics.

[133]  Michael Mabe,et al.  The growth and number of journals , 2003 .

[134]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 1 , 1998 .

[135]  Molly C Dougherty,et al.  Blinding in peer review: the preferences of reviewers for nursing journals. , 2008, Journal of advanced nursing.

[136]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[137]  Daniel S. Hamermesh,et al.  Facts and Myths about Refereeing , 1994 .

[138]  L. Rieseberg,et al.  No crisis in supply of peer reviewers , 2010, Nature.

[139]  Bo-Christer Björk,et al.  Scientific journal publishing: yearly volume and open access availability , 2009, Inf. Res..

[140]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[141]  J. R. Gilbert,et al.  Is there gender bias in JAMA's peer review process? , 1994, JAMA.

[142]  J Walsh,et al.  NSF and Its Critics in Congress: New Pressure on Peer Review. , 1975, Science.

[143]  Lutz Bornmann,et al.  Peer Review and Bibliometric: Potentials and Problems , 2011 .

[144]  David M. Schultz,et al.  REjEctIOn RAtES FOR jOURnAlS PUBlISHInG In tHE AtMOSPHERIc ScIEncES , 2010 .

[145]  Chandan K Sen Rebound peer review: a viable recourse for aggrieved authors? , 2012, Antioxidants & redox signaling.

[146]  Richard N. Zare,et al.  Interdisciplinary Research: From Belief to Reality , 1999, Science.

[147]  Stephen D. Gottfredson,et al.  Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. , 1978 .

[148]  J. Ioannidis Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. , 1998, JAMA.

[149]  David N. Laband,et al.  Favoritism versus Search for Good Papers: Empirical Evidence Regarding the Behavior of Journal Editors , 1994, Journal of Political Economy.

[150]  Fiona Godlee,et al.  Effect of blinding and unmasking on the quality of peer review , 1999, Journal of General Internal Medicine.

[151]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[152]  Ruth E. Duerr,et al.  Data Citation and Peer Review , 2010 .

[153]  E. Ernst,et al.  Reviewer bias. , 1992, Annals of internal medicine.

[154]  Juan Miguel Campanario,et al.  Rejecting highly cited papers: The views of scientists who encounter resistance to their discoveries from other scientists , 2007, J. Assoc. Inf. Sci. Technol..

[155]  Joseph Smith,et al.  Impact of blinded versus unblinded abstract review on scientific program content. , 2002, The Journal of urology.

[156]  J. Ziman Real Science: What It Is and What It Means , 2000 .

[157]  Lutz Bornmann,et al.  The potential and problems of peer evaluation in higher education and research , 2007 .

[158]  R. Steinpreis,et al.  The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study , 1999 .

[159]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[160]  Derek de Solla Price,et al.  A general theory of bibliometric and other cumulative advantage processes , 1976, J. Am. Soc. Inf. Sci..

[161]  J. Rust,et al.  Modern Psychometrics: The Science of Psychological Assessment , 1989 .

[162]  R. Whittaker,et al.  Journal review and gender equality: a critical comment on Budden et al. , 2008, Trends in ecology & evolution.

[163]  Lutz Bornmann,et al.  Potential sources of bias in research fellowship assessments: effects of university prestige and field of study , 2006 .

[164]  E Frank,et al.  Editors' requests of peer reviewers: a study and a proposal. , 1996, Preventive medicine.

[165]  Donald S. Rubenstein Pharmaceutical Ads in Annals , 1992 .

[166]  Stefan Hirschauer,et al.  Editorial Judgments , 2010 .

[167]  Simon Wessely,et al.  Peer review of grant applications: what do we know? , 1998, The Lancet.

[168]  Lutz Bornmann,et al.  Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition , 2009, J. Informetrics.

[169]  J. Brooks Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[170]  M. Biagioli,et al.  LIFE SCIENCES FORUM The Instability of Authorship: Credit and Responsibility in Contemporary Biomedicine , 1998 .

[171]  M. Kon,et al.  Who benefits from peer review? An analysis of the outcome of 100 requests for review by Plastic and Reconstructive Surgery. , 2006, Plastic and reconstructive surgery.

[172]  E Ernst,et al.  A randomized controlled study of reviewer bias against an unconventional therapy , 2000, Journal of the Royal Society of Medicine.

[173]  M. Biagioli From Book Censorship to Academic Peer Review , 2002 .

[174]  Ulf Sandström,et al.  Cognitive Bias in Peer Review : a New Approach , 2009 .

[175]  Richard J. C. Brown,et al.  Double anonymity in peer review within the chemistry periodicals community , 2007, Learn. Publ..

[176]  Steven Shapin,et al.  A Social History of Truth: Civility and Science in Seventeenth-Century England. , 1995 .

[177]  T. Gustafson The Controversy Over Peer Review , 1975, Science.

[178]  M. Mahoney Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[179]  Stevan Harnad,et al.  Peer Commentary on Peer Review: A Case Study in Scientific Quality Control , 1983 .

[180]  David Lane Double-blind review: easy to guess in specialist fields. , 2008, Nature.

[181]  Gary J. Becker The Peer-Review Process at JVIR , 1993 .

[182]  Lutz Bornmann,et al.  Do Author-Suggested Reviewers Rate Submissions More Favorably than Editor-Suggested Reviewers? A Study on Atmospheric Chemistry and Physics , 2010, PloS one.

[183]  D. Rennie,et al.  Publication bias in editorial decision making. , 2002, JAMA.