To intervene or not to intervene; is that the question? On the role of scientometrics in research evaluation

Recent high‐profile statements, criticisms, and boycotts organized against certain quantitative indicators (e.g., the DORA declaration) have brought misuses of performance metrics to the center of attention. A key concern captured in these movements is that the metrics appear to carry authority even where established agents of quality control have explicitly outlined limits to their validity and reliability as measurement tools. This raises a number of challenging questions for those readers of this journal who are implicated in questions of indicator “production” and, by extension, “effects.” In this opinion piece we wish to critically engage the question of how producers of indicators can come to terms with their role as (partly) responsible parties in the current age of evaluative bibliometrics. We do so through the illuminating case of the professional scientometrics community.

[1]  R. Andersen,et al.  The Fifth Branch: Science Advisors as Policymakers , 1991 .

[2]  Blaise Cronin,et al.  The sociological turn in information science , 2008, J. Inf. Sci..

[3]  Wolfgang Glänzel,et al.  The need for standards in bibliometric research and technology , 2005, Scientometrics.

[4]  Anthony F. J. van Raan Little scientometrics, big scientometrics ... and beyond , 2005, Scientometrics.

[5]  Thane Chambers Scholarly Metrics Under the Microscope: From Citation Analysis to Academic Auditing , 2015 .

[6]  Gemma Derrick,et al.  "A number you just can't get away from": Characteristics of Adoption and the Social Construction of Metric Use by Researchers , 2013 .

[7]  Yuan-Cheng Lai,et al.  Improving the Accuracy of , 2014 .

[8]  Cassidy R. Sugimoto,et al.  The Citation: From Culture to Infrastructure , 2014 .

[9]  Gualberto Buela-Casal,et al.  What do the scientists think about the impact factor? , 2012, Scientometrics.

[10]  D. Aksnes,et al.  Researchers’ perceptions of citations , 2009 .

[11]  E. Garfield Fortnightly Review: How can impact factors be improved? , 1996 .

[12]  Ludo Waltman,et al.  The inconsistency of the h-index , 2011, J. Assoc. Inf. Sci. Technol..

[13]  Henk F. Moed,et al.  Improving the accuracy of Institute for Scientific Information's journal impact factors , 1995 .

[14]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[15]  Erin A. Cech,et al.  How Economics Shapes Science , 2013 .

[16]  J. Sylvan Katz Bibliometric standards: Personal experience and lessons learned , 2005, Scientometrics.

[17]  Rodrigo Costas,et al.  The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level , 2007, J. Informetrics.

[18]  T. V. Leeuwen,et al.  Impact factors can mislead , 1996, Nature.

[19]  J. G. Frankfort,et al.  An Approach to University Science Policy: A New Research-Funding System. , 1980 .

[20]  Jerome K. Vanclay,et al.  Impact factor: outdated artefact or stepping-stone to journal certification? , 2011, Scientometrics.

[21]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[22]  Eugene Garfield,et al.  Is citation analysis a legitimate evaluation tool? , 2005, Scientometrics.

[23]  P. Wouters The citation culture , 1999 .

[24]  Ben R. Martin,et al.  Science and technology studies: exploring the knowledge base , 2011 .

[25]  David Pontille,et al.  La manufacture de l'évaluation scientifique: Algorithmes, jeux de données et outils bibliométriques , 2013 .

[26]  I. Ràfols,et al.  The dilemmas of performance indicators of individual researchers – An urgent debate in bibliometrics , 2013 .

[27]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[28]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[29]  H. Nowotny Democratising expertise and socially robust knowledge , 2003 .

[30]  S. Jasanoff,et al.  The Fifth Branch: Science Advisers as Policymakers. , 1991 .

[31]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems , 2004 .

[32]  A. F. J. van Raan,et al.  Handbook of quantitative studies of science and technology , 1988 .

[33]  E. Garfield The history and meaning of the journal impact factor. , 2006, JAMA.

[34]  Linda Butler,et al.  What Happens when Funding is Linked to Publication Counts , 2004 .