Looking for the impact of peer review: does count of funding acknowledgements really predict research impact?

A small number of studies have sought to establish that research papers with more funding acknowledgements achieve higher impact and have claimed that such a link exists because research supported by more funding bodies undergoes more peer review. In this paper, a test of this link is made using recently available data from the Web of Science, a source of bibliographic data that now includes funding acknowledgements. The analysis uses 3,596 papers from a single year, 2009, and a single journal, the Journal of Biological Chemistry. Analysis of this data using OLS regression and two ranks tests reveals the link between count of funding acknowledgements and high impact papers to be statistically significant, but weak. It is concluded that count of funding acknowledgements should not be considered a reliable indicator of research impact at this level. Relatedly, indicators based on assumptions that may hold true at one level of analysis may not be appropriate at other levels.

[1]  Stephen J. Bensman,et al.  Classification and powerlaws: The logarithmic transformation , 2006, J. Assoc. Inf. Sci. Technol..

[2]  T. V. Leeuwen,et al.  The use of combined bibliometric methods in research funding policy , 2001 .

[3]  S. Baldi.  Normative versus social constructivist processes in the allocation of citations : A network-analytic model , 1998 .

[4]  G. Lewison,et al.  Bibliometric methods for the evaluation of arthritis research. , 1999, Rheumatology.

[5]  Grant Lewison,et al.  Reviewers’ and editors’ perceptions of submitted manuscripts with different numbers of authors, addresses and funding sources , 1999, J. Inf. Sci..

[6]  Stephen J. Bensman,et al.  Classification and powerlaws: The logarithmic transformation: Research Articles , 2006 .

[7]  Dwijen Rangnekar,et al.  Acknowledged: analysing the bibliometric presence of the Multiple Sclerosis Society , 2005, Aslib Proc..

[8]  Grant Lewison,et al.  The evaluation of Russian cancer research , 2010 .

[9]  J. Ioannidis,et al.  Sources of funding for Nobel Prize‐winning work: public or private? , 2010, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[10]  L. Butler,et al.  Revisiting bibliometric issues using new empirical data , 2001 .

[11]  J. Rigby.  Systematic grant and funding body acknowledgement data for publications: new dimensions and new controversies for research policy and evaluation , 2011 .

[12]  Grant Lewison,et al.  The effect of funding on the outputs of biomedical research , 2006, Scientometrics.

[13]  Kevin W Boyack,et al.  Mapping knowledge domains: Characterizing PNAS , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Stefan Hornbostel,et al.  Third party funding of German universities. An indicator of research activity? , 2001, Scientometrics.

[15]  P. Shapira,et al.  Organizational and institutional influences on creativity in scientific research , 2009 .

[16]  Hans Stam,et al.  [High quality of research supported by the Netherlands Heart Foundation ('Nederlandse Hartstichting')]. , 2010, Nederlands tijdschrift voor geneeskunde.

[17]  Sam Wilson,et al.  What makes an article influential? Predicting impact in social and personality psychology , 2008, Scientometrics.

[18]  Jeremy M Berg.  A Nobel Lesson: The Grant Behind the Prize , 2008, Science.

[19]  Carl F Albrecht,et al.  A bibliometric analysis of research publications funded partially by the Cancer Association of South Africa (CANSA) during a 10-year period (1994–2003) , 2009 .

[20]  G Lewison,et al.  International gastroenterology research: subject areas, impact, and funding. , 2001, Gut.

[21]  Response from science , 1984, Cell.

[22]  J. Carl Barrett,et al.  Revamp the NTP bioassay. , 1995, Environmental health perspectives.

[23]  Éric Archambault,et al.  Bibliometrics as a Performance Measurement Tool for Research Evaluation: The Case of Research Funded by the National Cancer Institute of Canada , 2010 .

[24]  Grant Lewison.  Publications from the European community's biotechnology action programme (BAP): Multinationality, acknowledgement of support, and citations , 2005, Scientometrics.

[25]  L. Butler,et al.  The efficacy of different modes of funding research: perspectives from Australian data on the biological sciences , 1999 .

[26]  Miranda Lee Pao,et al.  On the relationship of funding and research publications , 2005, Scientometrics.

[27]  Gabriel Pinski,et al.  Structure of the Biomedical Literature , 1976, J. Am. Soc. Inf. Sci..

[28]  Ulf Sandström,et al.  Research quality and diversity of funding: A model for relating research money to output of research , 2009, Scientometrics.