Alternative metric indicators for funding scheme evaluations

Purpose – The purpose of this paper is to investigate the potential of altmetric and webometric indicators to aid with funding agencies’ evaluations of their funding schemes. Design/methodology/approach – This paper analyses a range of altmetric and webometric indicators in terms of suitability for funding scheme evaluations, compares them to traditional indicators and reports some statistics derived from a pilot study with Wellcome Trust-associated publications. Findings – Some alternative indicators have advantages to usefully complement scientometric data by reflecting a different type of impact or through being available before citation data. Research limitations/implications – The empirical part of the results is based on a single case study and does not give statistical evidence for the added value of any of the indicators. Practical implications – A few selected alternative indicators can be used by funding agencies as part of their funding scheme evaluations if they are processed in ways that enable comparisons between data sets. Their evidence value is only weak, however. Originality/value – This is the first analysis of altmetrics or webometrics from a funding scheme evaluation perspective.

[1]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[2]  Mike Thelwall,et al.  Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals , 2008, J. Assoc. Inf. Sci. Technol..

[3]  Mike Thelwall,et al.  Do highly cited researchers successfully use the social web? , 2014, Scientometrics.

[4]  Jenny Fry,et al.  Measuring researchers’ use of scholarly information through social bookmarking data: A case study of BibSonomy , 2012, J. Inf. Sci..

[5]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[6]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[7]  Martin Meyer,et al.  Academic patents as an indicator of useful research? A new approach to measure academic inventiveness , 2003 .

[8]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[9]  Massimo Franceschet,et al.  The first Italian research assessment exercise: A bibliometric perspective , 2009, J. Informetrics.

[10]  Mike Thelwall,et al.  How is research blogged? A content analysis approach , 2015, J. Assoc. Inf. Sci. Technol..

[11]  Mike Thelwall,et al.  Guideline references and academic citations as evidence of the clinical value of health research , 2016, J. Assoc. Inf. Sci. Technol..

[12]  Giovanni Abramo,et al.  The VQR, Italy's second national research assessment: Methodological failures and ranking distortions , 2015, J. Assoc. Inf. Sci. Technol..

[13]  Hartmut Raffler,et al.  Interim Evaluation of the Seventh Framework Programme: Report of the Expert Group , 2010 .

[14]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[15]  Gary Taubes,et al.  Bad Science: The Short Life and Weird Times of Cold Fusion , 1994 .

[16]  Liz Allen,et al.  Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact , 2014, PLoS biology.

[17]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[18]  Adam B. Jaffe,et al.  Building Programme Evaluation into the Design of Public Research‐Support Programmes , 2002 .

[19]  Mike Thelwall,et al.  Regression for citation data: An evaluation of different methods , 2014, J. Informetrics.

[20]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[21]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[22]  Mike Thelwall,et al.  Tweeting Links to Academic Articles , 2013 .

[23]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[24]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[25]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[26]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[27]  Mike Thelwall,et al.  The influence of time and discipline on the magnitude of correlations between citation counts and quality scores , 2015, J. Informetrics.

[28]  Danielle Moran,et al.  Using Twitter in university research, teaching and impact activities , 2011 .

[29]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 2: Social media metrics , 2015 .

[30]  R. Tijssen Global and domestic utilization of industrial relevant science: patent citation analysis of science-technology interactions and knowledge flows , 2001 .

[31]  Francis Narin,et al.  Patent bibliometrics , 2005, Scientometrics.

[32]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[33]  Mike Thelwall,et al.  Arts and humanities research evaluation: no metrics please, just data , 2015, J. Documentation.

[34]  Michel Zitt,et al.  The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay’s article 2011 , 2012, Scientometrics.

[35]  Mike Thelwall,et al.  Can the impact of non‐Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia , 2014, J. Assoc. Inf. Sci. Technol..

[36]  Alesia A. Zuccala,et al.  Comparing book citations in humanities journals to library holdings : scholarly use versus 'perceived cultural benefit' (RIP) , 2013 .

[37]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[38]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.

[39]  Peter Bond,et al.  Bad Science: The Short Life and Weird Times of Cold Fusion , 1993 .

[40]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[41]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[42]  Nicolás Robinson-García,et al.  Towards a Book Publishers Citation Reports. First approach using the Book Citation Index , 2012, Revista española de Documentación Científica.

[43]  M. Thelwall,et al.  Google Scholar citations and Google Web-URL citations: A multi-discipline exploratory analysis , 2007 .

[44]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[45]  Mike Thelwall,et al.  More precise methods for national research citation impact comparisons , 2015, J. Informetrics.

[46]  Mike Thelwall,et al.  National research impact indicators from Mendeley readers , 2015, J. Informetrics.

[47]  Stefanie Haustein,et al.  Exploring data quality and retrieval strategies for mendeley reader counts , 2015 .

[48]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web , 2015 .

[49]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[50]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[51]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[52]  A. Raan,et al.  A bibliometric analysis of six economics research groups: A comparison with peer review , 1993 .

[53]  Stacy Konkiel,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2016 .

[54]  Mike Thelwall,et al.  An automatic method for assessing the teaching impact of books from online academic syllabi , 2016, J. Assoc. Inf. Sci. Technol..

[55]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[56]  Mike Thelwall,et al.  Substance without citation: evaluating the online impact of grey literature , 2014, Scientometrics.

[57]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[58]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[59]  Mike Thelwall,et al.  An automatic method for extracting citations from Google Books , 2015, J. Assoc. Inf. Sci. Technol..

[60]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008 .

[61]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 3: books and non standard outputs , 2015 .