Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process

High publication demands and the low acceptance rate of peer review journals place the journal editors and their reviewers in a powerful position. Journal reviewers have a vital role not only in influencing the journal editor's publication decisions, but also in the very nature and direction of scientific research. Because of their influence in peer review outcomes, journal reviewers are aptly described as the “gate keepers of science.” In this article we describe several pitfalls that can impede reviewers' impartial judgement. These include such issues as confirmatory bias, the negative results bias (the file drawer problem), the Matthew effect, the Doctor Fox effect, and gender, race, theoretical orientation, and “political correctness.” We argue that procedures currently used by many professional journals, such as blind or masked review, may not completely alleviate the effects of these pitfalls. Instead, we suggest that increasing reviewers' awareness of the pitfalls, accountability, and vigilance can improve fairness in the peer review process. The ultimate responsibilities belong to the journal editors who are confronted with the difficult task of satisfying journal readers, contributors, reviewers, and owners. We recommend that the journal editors conduct periodic internal and external evaluations of their journals' peer review process and outcomes, with participation of reviewers, contributors, readers and owners.

[1]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[2]  Alfred de Grazia,et al.  The Scientific Reception System and Dr. Velikovsky , 1963 .

[3]  J. Bartko.  The Intraclass Correlation Coefficient as a Measure of Reliability , 1966, Psychological reports.

[4]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[5]  R. Lippman.  More on Community Mental Health Centers amendments of 1969. , 1971 .

[6]  J E Ware,et al.  The Doctor Fox Lecture: a paradigm of educational seduction. , 1973, Journal of medical education.

[7]  W. A. Scott,et al.  Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. , 1974 .

[8]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[9]  R G Williams,et al.  The Dr. Fox effect: a study of lecturer effectiveness and ratings of instruction. , 1975, Journal of medical education.

[10]  Duncan Lindsey,et al.  Distinction, achievement, and editorial board membership. , 1976 .

[11]  G. D. Gottfredson,et al.  Geographic distribution of U.S. psychologists: A human ecological analysis. , 1978 .

[12]  R. Crandall.  Interrater agreement on manuscript is not so badp. , 1978 .

[13]  Stephen D. Gottfredson,et al.  Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. , 1978 .

[14]  S. Scarr,et al.  The reliability of reviews for the American Psychologist. , 1978 .

[15]  M. Moore,et al.  Discrimination or Favoritism? Sex Bias in Book Reviews. , 1978 .

[16]  Judith A. Hall.  Author review of reviewers. , 1979 .

[17]  Andrew M. Colman,et al.  Editorial role in author- referee disagreements , 1979 .

[18]  Marley W. Watkins,et al.  Chance and interrater agreement on manuscripts. , 1979 .

[19]  Domenic V. Cicchetti,et al.  Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. , 1980 .

[20]  Donald W. King,et al.  Scientific journals in the United States: Their production, use, and economics , 1981 .

[21]  R. Over.  What is the source of bias in peer review? , 1982, Behavioral and Brain Sciences.

[22]  D. Horrobin.  Peer review: A philosophically faulty concept which is proving disastrous for science , 1982, Behavioral and Brain Sciences.

[23]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[24]  R. Crandall,et al.  Editorial responsibilities in manuscript review , 1982, Behavioral and Brain Sciences.

[25]  S. Lock,et al.  A difficult balance: editorial peer review in medicine continued. , 1985 .

[26]  J C Bailar,et al.  The need for a research agenda. , 1985, The New England journal of medicine.

[27]  J. Belsky.  Infant day care: A cause for concern? , 1986 .

[28]  Jeanne Marecek,et al.  The Meaning Of Difference: Gender Theory, Postmodernism, And Psychology , 1988 .

[29]  K. Dickersin.  The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[30]  R. Fletcher,et al.  The effects of blinding on the quality of peer review. A randomized trial. , 1990, JAMA.

[31]  C. Olson,et al.  Peer review of the biomedical literature. , 1990, The American journal of emergency medicine.

[32]  J. Marecek,et al.  Making a Difference: Psychology and the Construction of Gender , 1990 .

[33]  D. W. Sharp,et al.  What can and should be done to reduce publication bias? The perspective of an editor. , 1990, JAMA.

[34]  L. Silverstein,et al.  Transforming the debate about child care and maternal employment. , 1991, The American psychologist.

[35]  G H Guyatt,et al.  Agreement among reviewers of review articles. , 1991, Journal of clinical epidemiology.

[36]  D. Segrist,et al.  Sex bias in psychological research. Progress or complacency? , 1992, The American psychologist.

[37]  The future of medical journals , 1992 .

[38]  S. Riger.  Epistemological debates, feminist voices: Science, social values, and the study of women. , 1992 .

[39]  P. Riis.  New paradigms in journalology , 1992, Journal of internal medicine.

[40]  The National Family Wars. , 1993 .

[41]  M. Hojat.  The World Declaration of the Rights of the Child: Anticipated Challenges , 1993, Psychological reports.

[42]  David Popenoe,et al.  American Family Decline, 1960-1990: A Review and Appraisal. , 1993 .

[43]  Paul H. Rubin.  The Assault on the First Amendment: Public Choice and Political Correctness , 1994 .

[44]  J. R. Gilbert,et al.  Is there gender bias in JAMA's peer review process? , 1994, JAMA.

[45]  S. B. Friedman,et al.  The effects of blinding on acceptance of research papers by peer review. , 1994, JAMA.

[46]  M Nylenna,et al.  Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. , 1994, JAMA.

[47]  E. Lawson,et al.  Effect of institutional prestige on reviewers' recommendations and editorial decisions. , 1994, JAMA.

[48]  Political correctness and American academe , 1994 .

[49]  G. Loury.  Self-Censorship in Public Discourse , 1994 .

[50]  B. Sweitzer,et al.  How well does a journal's peer review process function? A survey of authors' opinions. , 1994, JAMA.

[51]  D G Altman,et al.  Is there a case for an international medical scientific press council? , 1994, JAMA.

[52]  D. Laband,et al.  A citation analysis of the impact of blinded peer review. , 1994, JAMA.

[53]  J. Kassirer,et al.  Peer review. Crude and understudied, but indispensable. , 1994, JAMA.

[54]  J. Hoover,et al.  The Political Correctness Controversy Revisited , 1995 .

[55]  A. Eagly.  Reflections on the Commenters' Views. , 1995 .

[56]  A. Eagly.  The science and politics of comparing women and men , 1995 .

[57]  R. Herrnstein,et al.  The bell curve : intelligence and class structure in American life , 1995 .

[58]  Jean-Pierre EN Pierie,et al.  Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde , 1996, The Lancet.

[59]  R. Horton.  A Physician’s Life , 1878, The Lancet.

[60]  R. Horton.  The Lancet's ombudsman , 1996, The Lancet.

[61]  M. Northridge,et al.  Editor's Note: Reviewing for the Journal , 1996 .

[62]  T. Albert.  Why bother with peer review? , 1997, The Lancet.

[63]  J. Stephenson.  Medical journals turn gaze inward to examine process of peer review. , 1997, JAMA.

[64]  L. Bero,et al.  Publication bias and research on passive smoking: comparison of published and unpublished studies. , 1998, JAMA.

[65]  R. Bauserman,et al.  A meta-analytic examination of assumed properties of child sexual abuse using college samples. , 1998, Psychological bulletin.

[66]  R. Wears,et al.  Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. , 1998, JAMA.

[67]  N. Black,et al.  What makes a good reviewer and a good review for a general medical journal? , 1998, JAMA.

[68]  D. Rennie,et al.  Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. , 1998, JAMA.

[69]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[70]  The journal ombudsperson: a step toward scientific press oversight. , 1998, JAMA.

[71]  J F Waeckerle,et al.  Reliability of editors' subjective quality ratings of peer reviews of manuscripts. , 1998, JAMA.

[72]  G. Wilkinson,et al.  Peer review and editorial decision-making , 1998, British Journal of Psychiatry.

[73]  D Rennie.  Peer review in Prague. , 1998, JAMA.

[74]  D. Rennie,et al.  Masking author identity in peer review: what factors influence masking success? PEER Investigators. , 1998, JAMA.

[75]  The Importance of Reviewers , 1999, Science.

[76]  N. Black,et al.  Effect of blinding and unmasking on the quality of peer review: a randomized trial. , 1998, JAMA.

[77]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[78]  R Smith,et al.  Opening up BMJ peer review , 1999, BMJ.

[79]  R. Rosenberg,et al.  Editorial governance of the Journal of the American Medical Association: a report. , 1999, JAMA.

[80]  S Goldbeck-Wood,et al.  Evidence on peer review—scientific quality control or smokescreen? , 1999, BMJ.

[81]  M. Angell.  The Journal and its owner--resolving the crisis. , 1999, The New England journal of medicine.

[82]  J. Drazen,et al.  Tracking the peer-review process. , 2000, The New England journal of medicine.

[83]  C. Deangelis,et al.  Thanking Authors, Peer Reviewers, and Readers—Constancy in a Time of Change , 2000 .

[84]  Sponsorship, authorship, and accountability. , 2001, The Western journal of medicine.

[85]  Weathering a political storm. A contextual perspective on a psychological research controversy. , 2002 .

[86]  When worlds collide. Social science, politics, and the Rind et al. (1998). Child sexual abuse meta-analysis. , 2002 .

[87]  Donald Kennedy.  To Publish or Not to Publish , 2002, Science.

[88]  M. Hojat,et al.  Effects of Reviewers' Gender on Assessments of a Gender-Related Standardized Manuscript , 2003, Teaching and learning in medicine.

[89]  M. Mahoney.  Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.