Past performance, peer review and project selection: a case study in the social and behavioral sciences

Does past performance influence success in grant applications? We tested whether the decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of applicants in publications and citations, and with the results of the Council's peer reviews. The Council proves successful in distinguishing grant applicants with above-average from below-average performance, but within the former group there was no correlation between past performance and receiving a grant. When comparing the best-performing researchers who were denied funding with those who received it, the rejected researchers significantly outperformed the funded ones. The best rejected proposals score on average as high on the outcomes of the peer-review process as the accepted proposals. The Council successfully corrected for gender effects during the selection process. We explain why these findings may apply beyond this case. However, if research councils are not able to select the ‘best’ researchers, perhaps they should reconsider their mission. We discuss the role of research councils in the science system in terms of variation, innovation and quality control. Copyright , Beech Tree Publishing.

[1]  Peter Taylor,et al.  Citation Statistics , 2009, ArXiv.

[2]  P. Farber.  Journals Under Threat: A Joint Response from HSTM Editors , 2009 .

[3]  Charles Oppenheim,et al.  Out With the Old and in With the New: the RAE, Bibliometrics and the New REF , 2008, J. Libr. Inf. Sci..

[4]  Linda Butler,et al.  Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework , 2008 .

[5]  Ulf Sandström,et al.  Persistent nepotism in peer-review , 2008, Scientometrics.

[6]  Loet Leydesdorff,et al.  Caveats for the Use of Citation Indicators in Research and Journal Evaluations , 2008, J. Assoc. Inf. Sci. Technol..

[7]  B. Kehm.  Hochschule im Wandel: Die Universität als Forschungsgegenstand , 2008 .

[8]  Stefan Hornbostel,et al.  Postdocs in Deutschland: Evaluation des Emmy Noether-Programms , 2008 .

[9]  Loet Leydesdorff,et al.  Past performance as predictor of successful grant applications: A case study. Report to the board of the Netherlands Social Science Research council (MaGW/NWO) , 2009, 0911.3085.

[10]  Paul Nightingale,et al.  Peer review and the relevance gap: Ten suggestions for policy-makers , 2007 .

[11]  L. Butler,et al.  Assessing university research: A plea for a balanced approach , 2007 .

[12]  Henk F. Moed,et al.  The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review , 2007 .

[13]  C. Donovan.  Introduction: Future pathways for science policy and research assessment: Metrics vs peer review, quality vs impact , 2007 .

[14]  P. Lawrence.  The mismeasurement of science , 2007, Current Biology.

[15]  Katharine Barker,et al.  The UK Research Assessment Exercise: the evolution of a national research evaluation system , 2007 .

[16]  L. Bornmann,et al.  Gender differences in grant peer review: A meta-analysis , 2007, J. Informetrics.

[17]  Lutz Bornmann,et al.  Potential sources of bias in research fellowship assessments: effects of university prestige and field of study , 2006 .

[18]  Rickard Danell,et al.  The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden , 2006 .

[19]  Lutz Bornmann,et al.  Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants , 2006, Scientometrics.

[20]  Gloria Origgi,et al.  Introduction to a special issue on the assessment of interdisciplinary research , 2006 .

[21]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[22]  D. Aksnes,et al.  Peer reviews and bibliometric indicators: a comparative study at a Norwegian university , 2004 .

[23]  Liv Langfeldt,et al.  Expert panels evaluating research: decision-making and sources of bias , 2004 .

[24]  Stefan Timmermans,et al.  Research Groups in Dutch Sociology , 2003 .

[25]  Gerhard Fröhlich,et al.  Anonyme Kritik: Peer Review auf dem Prüfstand der Wissenschaftsforschung , 2002 .

[26]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[27]  Simon Wessely,et al.  Peer review of grant applications: what do we know? , 1998, The Lancet.

[28]  D. Cicchetti.  The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation , 1991, Behavioral and Brain Sciences.

[29]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[30]  B. Martin,et al.  Foresight in Science: Picking the Winners , 1984 .

[31]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[32]  M. Mulkay.  The Mediating Role of the Scientific Elite , 1976 .

[33]  V. L. Allen,et al.  Creativity and conformity. , 1968, Journal of personality.

[34]  C. Moustakas,et al.  Creativity and conformity , 1967 .