Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model

Peer review serves a gatekeeper role, the final arbiter of what is valued in academia, but is widely criticized in terms of potential biases—particularly in relation to gender. In this substantive-methodological synergy, we demonstrate methodological and multilevel statistical approaches to testing a null hypothesis model in relation to the effect of researcher gender on peer reviews of grant proposals, based on 10,023 reviews by 6233 external assessors of 2331 proposals from social science, humanities, and science disciplines. Utilizing multilevel cross-classified models, we show that support for the null hypothesis model positing researcher gender has no significant effect on proposal outcomes. Furthermore, these non-effects of gender generalize over assessor gender (contrary to a matching hypothesis), discipline, assessors chosen by the researchers themselves compared to those chosen by the funding agency, and country of the assessor. Given the large, diverse sample, the powerful statistical analyses, and support for generalizability, these results – coupled with findings from previous research – offer strong support for the null hypothesis model of no gender differences in peer reviews of grant proposals.

[1]  D. Terry,et al.  Effects of work stress on psychological well-being and job satisfaction: The stress-buffering role of social support , 1993 .

[2]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[3]  James Hartley,et al.  The Psychology of Written Communication , 1981 .

[4]  J. Bentley,et al.  Gender Differences in the Careers of Academic Scientists and Engineers: A Literature Review. Special Report. , 2003 .

[5]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[6]  Kenneth A. Feldman,et al.  College students' views of male and female college teachers: Part I—Evidence from the social laboratory and experiments , 1992 .

[7]  Lutz Bornmann,et al.  Do women have less success in peer review , 2009 .

[8]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[9]  Kenneth A. Feldman,et al.  College students' views of male and female college teachers: Part II—Evidence from students' evaluations of their classroom teachers , 1993 .

[10]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[11]  Stanley Wasserman,et al.  A Handbook for Data Analysis in the Behavioral Sciences:@@@(Vol. 1) Statistical Issues@@@(Vol. 2) Methodological Issues. , 1994 .

[12]  J. Rodgers.  The epistemology of mathematical and statistical modeling: a quiet methodological revolution. , 2010, The American psychologist.

[13]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[14]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[15]  Harris Cooper,et al.  The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. , 2009, Psychological methods.

[16]  L. Bornmann,et al.  Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches , 2009 .

[17]  Nigel W. Bond,et al.  Peer Review in the Funding of Research in Higher Education: The Australian Experience , 2001 .

[18]  P. Abrami,et al.  Students' Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research , 1987 .

[19]  Jonathan R. Cole,et al.  Fair Science: Women in the Scientific Community , 1987 .

[20]  Harriet Zuckerman,et al.  Stratification in American Science , 1970 .

[21]  Ulf Sandström,et al.  Persistent nepotism in peer-review , 2008, Scientometrics.

[22]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[23]  Nigel W. Bond,et al.  A new reader trial approach to peer review in funding research grants: An Australian experiment , 2006, Scientometrics.

[24]  M. Mahoney,et al.  Open exchange and epistemic progress. , 1985 .

[25]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[26]  Contents , 1997, Current Opinion in Neurobiology.

[27]  Andrew J. Martin,et al.  A multilevel perspective on gender in classroom motivation and climate: Potential benefits of male teachers for boys? , 2008 .

[28]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[29]  D. Cicchetti.  The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation , 1991, Behavioral and Brain Sciences.

[30]  H. Zuckerman,et al.  The Outer Circle: Women in the Scientific Community , 1991 .

[31]  G. William Walster,et al.  Physical attractiveness and dating choice: A test of the matching hypothesis☆ , 1971 .

[32]  D D Fanestil.  Publish and perish? , 1999, Journal of the American Society of Nephrology : JASN.

[33]  Jonathan R. Cole,et al.  Fair Science: Women in the Scientific Community. , 1982 .

[34]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[35]  Daryl E. Chubin,et al.  Grants Peer Review in Theory and Practice , 1994 .

[36]  Matthijs Kalmijn,et al.  Assortative Mating by Cultural and Economic Occupational Status , 1994, American Journal of Sociology.

[37]  E. Johansson,et al.  Gender bias in female physician assessments. Women considered better suited for qualitative research , 2002, Scandinavian journal of primary health care.

[38]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[39]  Nigel W. Bond,et al.  Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid , 2007 .

[40]  Risto Lethonen.  Multilevel Statistical Models (3rd ed.) , 2005 .

[41]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[42]  Raymond P. Perry,et al.  The Scholarship of Teaching and Learning in Higher Education: An Evidence-Based Perspective , 2007 .

[43]  A. Tversky.  Features of Similarity , 1977 .

[44]  H. Marsh,et al.  Improving the Peer-review Process for Grant Applications , 2022 .

[45]  Lutz Bornmann,et al.  Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions , 2005, Scientometrics.

[46]  H. Marsh,et al.  Applications of latent-variable models in educational psychology: The need for methodological-substantive synergies. , 2007 .

[47]  William J Browne,et al.  MCMC Estimation in MLwiN (Version 2.13) Centre for Multilevel Modelling, University of Bristol , 2009 .

[48]  Samuel Ball,et al.  Reflections on the peer review process , 1991, Behavioral and Brain Sciences.

[49]  Lutz Bornmann,et al.  Gender differences in grant peer review: A meta-analysis , 2007, J. Informetrics.

[50]  Robert Perloff,et al.  IMPROVING MANUSCRIPT EVALUATION PROCEDURES , 1972 .

[51]  J. Hyde,et al.  The Gender Similarities Hypothesis , 2005 .

[52]  Gerd Gigerenzer,et al.  The superego, the ego, and the id in statistical reasoning , 1993 .

[53]  C. Wennerås,et al.  A chair of one's own , 2000, Nature.

[54]  R. P. McDonald,et al.  Goodness-of-fit indexes in confirmatory factor analysis : The effect of sample size , 1988 .

[55]  R. Merton,et al.  Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system , 1971 .