National research impact indicators from Mendeley readers

National research impact indicators derived from citation counts are used by governments to help assess their national research performance and to identify the effect of funding or policy changes. Citation counts lag research by several years, however, and so their information is somewhat out of date. Some of this lag can be avoided by using readership counts from the social reference sharing site Mendeley because these accumulate more quickly than citations. This article introduces a method to calculate national research impact indicators from Mendeley, using citation counts from older time periods to partially compensate for international biases in Mendeley readership. A refinement to accommodate recent national changes in Mendeley uptake makes little difference, despite being theoretically more accurate. The Mendeley patterns using the methods broadly reflect the results from similar calculations with citations and seem to reflect impact trends about a year earlier. Nevertheless, the reasons for the differences between the indicators from the two data sources are unclear.

[1]  Martin Meyer,et al.  What is Special about Patent Citations? Differences between Scientific and Patent Citations , 2000, Scientometrics.

[2]  Philip M. Davis,et al.  Cited Half-Life of the Journal Literature , 2015, ArXiv.

[3]  Dag W. Aksnes,et al.  Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods , 2012, J. Informetrics.

[4]  Henk F. Moed,et al.  Mapping of science by combined co-citation and word analysis, I. Structural aspects , 1991, J. Am. Soc. Inf. Sci..

[5]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[6]  Ludo Waltman,et al.  Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison , 2012, Scientometrics.

[7]  Frans Albarillo,et al.  Language in Social Science Databases: English Versus Non-English Articles in JSTOR and Scopus , 2014 .

[8]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[9]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[10]  Claudio Castellano,et al.  Field-normalized impact factors (IFs): A comparison of rescaling and fractionally counted IFs , 2013, J. Assoc. Inf. Sci. Technol..

[11]  Christina Courtright,et al.  Context in information behavior research , 2007 .

[12]  Mike Thelwall,et al.  Do highly cited researchers successfully use the social web? , 2014, Scientometrics.

[13]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[14]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[15]  Mike Thelwall,et al.  Guideline references and academic citations as evidence of the clinical value of health research , 2016, J. Assoc. Inf. Sci. Technol..

[16]  Xu Zhang,et al.  Influences of counting methods on country rankings: a perspective from patent analysis , 2013, Scientometrics.

[17]  Jie Li,et al.  Citation Analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar , 2010 .

[18]  Industrial Strategy,et al.  International comparative performance of the UK research base , 2012 .

[19]  Mike Thelwall,et al.  A combined bibliometric indicator to predict article impact , 2011, Inf. Process. Manag..

[20]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[21]  Mike Thelwall,et al.  Alphabetization and the skewing of first authorship towards last names early in the alphabet , 2013, J. Informetrics.

[22]  Chen Xiaoguan.  The UK: The Leader of International Research Base—Based on International Comparative Performance of the UK Research Base-2013 , 2015 .

[23]  Scott B. Weingart,et al.  Tweeting Links to Academic Articles 1 , 2013 .

[24]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[25]  Mu-Hsuan Huang,et al.  Counting methods, country rank changes, and counting inflation in the assessment of national research productivity and impact , 2011, J. Assoc. Inf. Sci. Technol..

[26]  Mike Thelwall,et al.  An automatic method for assessing the teaching impact of books from online academic syllabi , 2016, J. Assoc. Inf. Sci. Technol..

[27]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[28]  Vincent Larivière,et al.  Long-term variations in the aging of scientific literature: From exponential growth to steady-state science (1900-2004) , 2008, J. Assoc. Inf. Sci. Technol..

[29]  Fereshteh Didegah,et al.  A Comparison of Web of Science and Scopus for Iranian Publications and Citation Impact , 2013 .

[30]  Pedro Albarrán,et al.  A comparison of the scientific performance of the U.S. and the European Union at the turn of the XXI century , 2009 .

[31]  Vincent Larivière,et al.  Benchmarking scientific output in the social sciences and humanities: The limits of existing databases , 2006, Scientometrics.

[32]  Vincent Larivière,et al.  Conference proceedings as a source of scientific information: A bibliometric analysis , 2008, J. Assoc. Inf. Sci. Technol..

[33]  Mike Thelwall,et al.  Mendeley readership counts: An investigation of temporal and disciplinary differences , 2016, J. Assoc. Inf. Sci. Technol..

[34]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[35]  Michel Zitt,et al.  Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation , 2005, Scientometrics.

[36]  Joshua S. Gans,et al.  First Author Conditions , 1998 .

[37]  Masood Fooladi,et al.  A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases , 2013, ArXiv.

[38]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[39]  J. Ravetz.  Sociology of Science , 1972, Nature.

[40]  Henk F. Moed,et al.  First evidence of serious language-bias in the use of citation analysis for the evaluation of national science systems , 2000 .

[41]  Giovanni Abramo,et al.  The VQR, Italy's second national research assessment: Methodological failures and ranking distortions , 2015, J. Assoc. Inf. Sci. Technol..

[42]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[43]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[44]  M. Trajtenberg.  A Penny for Your Quotes : Patent Citations and the Value of Innovations , 1990 .

[45]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[46]  Michel Zitt,et al.  Modifying the journal impact factor by fractional citation weighting: The audience factor , 2008, J. Assoc. Inf. Sci. Technol..

[47]  Peter Ingwersen,et al.  The calculation of web impact factors , 1998, J. Documentation.

[48]  Vincent Larivière,et al.  Tweets vs. Mendeley readers: How do these two social media metrics differ? , 2014, it Inf. Technol..

[49]  Wolfgang Glänzel,et al.  A Bibliometric Study of Reference Literature in the Sciences and Social Sciences , 1999, Inf. Process. Manag..

[50]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[51]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1975, Journal for the Scientific Study of Religion.

[52]  Charles Oppenheim,et al.  Highly cited old papers and the reasons why they continue to be cited , 1978, J. Am. Soc. Inf. Sci..

[53]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[54]  Thed N. van Leeuwen,et al.  Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance , 2001, Scientometrics.

[55]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[56]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[57]  Henk F. Moed,et al.  Statistical relationships between downloads and citations at the level of individual documents within a single journal: Book Reviews , 2005 .

[58]  Ana Marušić,et al.  A Systematic Review of Research on the Meaning, Ethics and Practices of Authorship across Scholarly Disciplines , 2011, PloS one.

[59]  Henk F. Moed,et al.  Statistical relationships between downloads and citations at the level of individual documents within a single journal , 2005, J. Assoc. Inf. Sci. Technol..

[60]  C. Lee Giles,et al.  Scholarly publishing in the Internet age: a citation analysis of computer science literature , 2001, Inf. Process. Manag..

[61]  Jonathan Adams,et al.  Early citation counts correlate with accumulated impact , 2005, Scientometrics.

[62]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[63]  A. F. J. VAN RAAN,et al.  In matters of quantitative studies of science the fault of theorists is offering too little and asking too much , 1998, Scientometrics.

[64]  Vincent Larivière,et al.  Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter , 2014, J. Assoc. Inf. Sci. Technol..

[65]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[66]  Alfred E. Hartemink,et al.  Citations and the h index of soil researchers and journals in the Web of Science, Scopus, and Google Scholar , 2013, PeerJ.

[67]  Stefanie Haustein,et al.  Exploring data quality and retrieval strategies for mendeley reader counts , 2015 .

[68]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[69]  Mike Thelwall,et al.  Tweeting Links to Academic Articles , 2013 .

[70]  Juan Miguel Campanario,et al.  Coverage, field specialisation and the impact of scientific publishers indexed in the Book Citation Index , 2013, Online Inf. Rev..

[71]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[72]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[73]  Ludo Waltman,et al.  Field-Normalized Citation Impact Indicators and the Choice of an Appropriate Counting Method , 2015, ISSI.

[74]  Daryl E. Chubin,et al.  Content Analysis of References: Adjunct or Alternative to Citation Counting? , 1975 .

[75]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[76]  Henk F. Moed,et al.  Mapping of Science by Combined Co-Citation and Word Analysis. I. Structural Aspects , 1991 .

[77]  Henk F. Moed,et al.  Comparing bibliometric country-by-country rankings derived from the Web of Science and Scopus: the effect of poorly cited journals in oncology , 2009, J. Inf. Sci..

[78]  Jiang Li,et al.  Chinese-language articles are not biased in citations: Evidences from Chinese-English bilingual journals in Scopus and Web of Science , 2014, J. Informetrics.