What do we know about grant peer review in the health sciences?

Background: Peer review decisions award an estimated >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members. Conclusions: We suggest funders should acknowledge, assess and analyse the uncertainty around peer review, even using reviewers’ uncertainty as an input to funding decisions. Funders could consider a lottery element in some parts of their funding allocation process, to reduce both burden and bias, and allow better evaluation of decision processes. Alternatively, the distribution of scores from different reviewers could be better utilised as a possible way to identify novel, innovative research. Above all, there is a need for open, transparent experimentation and evaluation of different ways to fund research. This also requires more openness across the wider scientific community to support such investigations, acknowledging the lack of evidence about the primacy of the current system and the impossibility of achieving perfection.

[1]  Pat Bazeley,et al.  Defining 'Early Career' in Research , 2003 .

[2]  Michael S Lauer,et al.  Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute. , 2015, Circulation research.

[3]  S. Wooding,et al.  The answer is 17 years, what is the question: understanding time lags in translational research , 2011, Journal of the Royal Society of Medicine.

[4]  J. Mervis U.S. RESEARCH MANAGEMENT. NSF tries two-step review, drawing praise--and darts. , 2016, Science.

[5]  Science funding. NSF's 'Big Pitch' tests anonymized grant reviews. , 2012, Science.

[6]  A. Casadevall,et al.  NIH peer review percentile scores are poorly predictive of grant productivity , 2016, eLife.

[7]  Soogwan Doh,et al.  Impact of Alumni Connections on Peer Review Ratings and Selection Success Rate in National Research , 2017 .

[8]  Anssi Auvinen,et al.  Panel discussion does not improve reliability of peer review for medical research grant proposals. , 2012, Journal of clinical epidemiology.

[9]  Spiridon Cristina Monica,et al.  The Delphi process: a solution for reviewing novel grant applications , 2010, International journal of general medicine.

[10]  N. Graves,et al.  ‘Are you siding with a personality or the grant proposal?’: observations on how peer review panels function , 2017, Research Integrity and Peer Review.

[11]  L. Costello,et al.  Perspective: Is NIH Funding the “Best Science by the Best Scientists”? A Critique of the NIH R01 Research Grant Review Policies , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[12]  G. Guyatt,et al.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , 2008, BMJ : British Medical Journal.

[13]  Darko Hren,et al.  Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research , 2015, PloS one.

[14]  Corinne Alberti,et al.  Non-Financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in France , 2012, PloS one.

[15]  Bruno Giraudeau,et al.  Peer Review of Grant Applications: A Simple Method to Identify Proposals with Discordant Reviews , 2011, PloS one.

[16]  N. Danthi,et al.  Citation impact of NHLBI R01 grants funded through the American Recovery and Reinvestment Act as compared to R01 grants funded through a standard payline. , 2015, Circulation research.

[17]  Funding Science by Lottery , 2015 .

[18]  Nicholas Graves,et al.  On the time spent preparing grant proposals: an observational study of Australian researchers , 2013, BMJ Open.

[19]  Veronica Mansilla Assessing expert interdisciplinary work at the frontier: an empirical exploration , 2006 .

[20]  Nicholas Graves,et al.  Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel , 2011, BMJ : British Medical Journal.

[21]  Adrian G Barnett,et al.  A randomized trial of fellowships for early career researchers finds a high reliability in funding decisions. , 2016, Journal of clinical epidemiology.

[22]  Terttu Luukkonen,et al.  Conservatism and risk-taking in peer review: Emerging ERC practices , 2012 .

[23]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[24]  A. Palermo,et al.  Community engagement in research: frameworks for education and peer review. , 2010, American journal of public health.

[25]  Nicholas Graves,et al.  Using simplified peer review processes to fund research: a prospective study , 2015, BMJ Open.

[26]  S. Roorda The Real Cost of the NSERC Peer Review is Less than 5% of a Proposed Baseline Grant , 2009, Accountability in research.

[27]  Molly Carnes,et al.  A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques From Investigators at One Institution , 2015, Academic medicine : journal of the Association of American Medical Colleges.

[28]  Wouter Steenbeek,et al.  No evidence that gender contributes to personal research funding success in The Netherlands: A reaction to van der Lee and Ellemers , 2015, Proceedings of the National Academy of Sciences.

[29]  Liv Langfeldt,et al.  The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments , 2006 .

[30]  Patrick E. McKnight,et al.  Grant Peer Review: Improving Inter-Rater Reliability with Training , 2015, PloS one.

[31]  Rustum Roy,et al.  Funding Science: The Real Defects of Peer Review and An Alternative To It , 1985 .

[32]  Michael R Doran,et al.  Multimedia: a necessary step in the evolution of research funding applications. , 2014, Trends in biochemical sciences.

[33]  C Hodgson How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. , 1997, Journal of clinical epidemiology.

[34]  Liv Langfeldt,et al.  The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome , 2001, Peer review in an Era of Evaluation.

[35]  John Bohannon,et al.  National Science Foundation. Meeting for peer review at a resort that's virtually free. , 2011, Science.

[36]  Harold Varmus,et al.  Rescuing US biomedical research from its systemic flaws , 2014, Proceedings of the National Academy of Sciences.

[37]  Sara Schroter,et al.  Surveys of current status in biomedical science grant review: funding organisations' and grant reviewers' perspectives , 2010, BMC medicine.

[38]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[39]  Peter van den Besselaar,et al.  Early career grants, performance, and careers: A study on predictive validity of grant decisions , 2015, J. Informetrics.

[40]  J. R. Cole,et al.  Chance and consensus in peer review. , 1981, Science.

[41]  Bryan J. Poulin,et al.  Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant , 2009, Accountability in research.

[42]  Nicholas Graves,et al.  The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers , 2014, BMJ Open.

[43]  Laurel L. Haak,et al.  Race, Ethnicity, and NIH Research Awards , 2011, Science.

[44]  L. Stanton,et al.  Features of successful bids for funding of applied health research: a cohort study , 2014, Health Research Policy and Systems.

[45]  A. Casadevall,et al.  Reforming Science: Structural Reforms , 2011, Infection and Immunity.

[46]  A. Spiegel Commentary: new guidelines for NIH peer review: improving the system or undermining it? , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[47]  M. Lauer,et al.  Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants , 2015, Molecular Psychiatry.

[48]  N. Graves,et al.  Streamlined research funding using short proposals and accelerated peer review: an observational study , 2015, BMC Health Services Research.

[49]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[50]  Lutz Bornmann,et al.  Gender differences in grant peer review: A meta-analysis , 2007, J. Informetrics.

[51]  Nigel W. Bond,et al.  Peer Review in the Funding of Research in Higher Education: The Australian Experience , 2001 .

[52]  Qi Wang,et al.  Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology , 2015 .

[53]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[54]  Carole J. Lee Commensuration Bias in Peer Review , 2015, Philosophy of Science.

[55]  A. Casadevall,et al.  NIH Peer Review Reform—Change We Need, or Lipstick on a Pig? , 2009, Infection and Immunity.

[56]  Lutz Bornmann,et al.  Latent Markov modeling applied to grant peer review , 2006, J. Informetrics.

[57]  L. Bromham,et al.  Interdisciplinary research has consistently lower funding success , 2016, Nature.

[58]  Jonathan D. Linton,et al.  Improving the Peer review process: Capturing more information and enabling high-risk/high-return research , 2016 .

[59]  Lutz Bornmann,et al.  Does Gender Matter in Grant Peer Review? , 2012, Zeitschrift fur Psychologie.

[60]  Johan Bollen,et al.  An efficient system to fund science: from proposal review to peer-to-peer distributions , 2016, Scientometrics.

[61]  Irwin Feller,et al.  Multiple actors, multiple settings, multiple criteria: issues in assessing interdisciplinary research , 2006 .

[62]  R. Tamblyn,et al.  Health services and policy research in the first decade at the Canadian Institutes of Health Research. , 2016, CMAJ open.

[63]  Alex F. DeNoble,et al.  Review panel consensus and post-decision commercial performance: a study of early stage technologies , 2010 .

[64]  Peter van den Besselaar,et al.  Selection committee membership: Service or self-service , 2012, J. Informetrics.

[65]  Stephen A. Gallo,et al.  Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes , 2013, PloS one.

[66]  Steven Wooding,et al.  How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice , 2015, Health Research Policy and Systems.

[67]  A. Barnett,et al.  Funding by Lottery: Political Problems and Research Opportunities , 2016, mBio.

[68]  A. Kopstein,et al.  An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH , 2010, PloS one.

[69]  Nigel W. Bond,et al.  Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model , 2011, J. Informetrics.

[70]  N. Ellemers,et al.  Gender contributes to personal research funding success in The Netherlands , 2015, Proceedings of the National Academy of Sciences.

[71]  E. Larson,et al.  Time and costs of preparing and submitting an NIH grant application at a school of nursing. , 2015, Nursing outlook.

[72]  Joon Koh,et al.  The Bias of Sighted Reviewers in Research Proposal Evaluation: A Comparative Analysis of Blind and Open Review in Korea , 2000, Scientometrics.

[73]  Theodore Eugene Day,et al.  The big consequences of small biases: A simulation of peer review , 2015 .

[74]  Raymond E Spier,et al.  Peer review and innovation , 2002, Science and engineering ethics.

[75]  R. Green,et al.  Institutionalized biases in the award of research grants: a preliminary analysis revisiting the principle of accumulative advantage , 2004 .

[76]  M. D. Lindner,et al.  NIH Peer Review , 2015, The American journal of evaluation.

[77]  Ellie Ehrenfeld,et al.  NIH peer review of grant applications for clinical research. , 2004, JAMA.

[78]  Lutz Bornmann,et al.  Potential sources of bias in research fellowship assessments: effects of university prestige and field of study , 2006 .

[79]  Alan L. Porter,et al.  Peer Review of Interdisciplinary Research Proposals , 1985 .

[80]  R. R. Snell,et al.  Menage a Quoi? Optimal Number of Peer Reviewers , 2015, PloS one.

[81]  J. Kerner,et al.  Networking and knowledge exchange to promote the formation of transdisciplinary coalitions and levels of agreement among transdisciplinary peer reviewers. , 2013, Journal of public health management and practice : JPHMP.

[82]  S. Ceci,et al.  Understanding current causes of women's underrepresentation in science , 2011, Proceedings of the National Academy of Sciences.

[83]  Harold Maurice Collins,et al.  New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System , 1991 .

[84]  Martin Reinhart,et al.  Peer review of grant applications in biology and medicine. Reliability, fairness, and validity , 2009, Scientometrics.

[85]  Arturo Casadevall,et al.  Research Funding: the Case for a Modified Lottery , 2016, mBio.

[86]  Michèle Lamont,et al.  What is Originality in the Humanities and the Social Sciences? , 2004 .

[87]  S. Wooding,et al.  Evaluating Grant Peer Review in the Health Sciences: A review of the literature , 2009 .

[88]  F. Tubach,et al.  Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices , 2012, PloS one.

[89]  M. Lauer,et al.  Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants , 2014, Circulation research.

[90]  Yiling Chen,et al.  Using prediction markets to forecast research evaluations , 2015, Royal Society Open Science.

[91]  Christoph Bartneck,et al.  The consequences of competition: simulating the effects of research grant allocation strategies , 2016, Scientometrics.

[92]  Is peer review useful in assessing research proposals in Indigenous health? A case study , 2009, Health research policy and systems.

[93]  Stephen A. Gallo,et al.  The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications , 2016, PloS one.

[94]  Toni Scarpa Peer Review at NIH , 2006, Science.

[95]  Karim R. Lakhani,et al.  Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science , 2016, Manag. Sci..

[96]  Christopher W. Belter,et al.  Bibliometric indicators: opportunities and limits. , 2015, Journal of the Medical Library Association : JMLA.

[97]  M. Lauer,et al.  Percentile Ranking and Citation Impact of a Large Cohort of National Heart, Lung, and Blood Institute–Funded Cardiovascular R01 Grants , 2014, Circulation research.

[98]  Molly Carnes,et al.  Threats to objectivity in peer review: the case of gender. , 2014, Trends in pharmacological sciences.