Cross-metric compatability and inconsistencies of altmetrics

Metrics like the number of tweets or Mendeley readers are currently discussed as an alternative to evaluate research. These alternative metrics (altmetrics) still need to be evaluated in order to fully understand their meaning, their benefits and limitations. While several preceding studies concentrate on correlations of altmetrics with classical measures like citations, this study aims at investigating metric-compatibility within altmetrics. For this purpose, 5000 journal articles from six disciplines have been analyzed regarding their metrics with the help of the aggregators PlumX and Altmetric.com. For this set, the highest numbers of events have been recognized regarding Mendeley readers, followed by Twitter and Facebook mentions. Thereby variations considering the aggregators could be observed. Intra-correlations between the metrics across one aggregator have been calculated, as well as inter-correlations for the corresponding metrics across the aggregators. For both aggregators, low to medium intra-correlations could be calculated which shows the diversity of the different metrics. Regarding inter-correlations, PlumX and Altmetric.com are highly consistent concerning Mendeley readers ($$r=0.97$$r=0.97) and Wikipedia mentions ($$r=0.82$$r=0.82), whereas the consistency concerning Twitter ($$r=0.49$$r=0.49), blogs ($$r=0.46$$r=0.46) and Reddit ($$r=0.41$$r=0.41) is on a moderate level. The sources Facebook ($$r=0.29$$r=0.29), Google+ ($$r=0.28$$r=0.28) and News ($$r=0.11$$r=0.11) show only low correlations.

[1]  Stacy Konkiel.  Grand challenges in altmetrics: heterogeneity, data quality and dependencies , 2017 .

[2]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[3]  Olle Persson.  'Citation Indexes for Science' - A 50 year citation history , 2005 .

[4]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[5]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[6]  Katrin Weller.  Social Media and Altmetrics: An Overview of Current Alternative Approaches to Measuring Scholarly Impact , 2015 .

[7]  Judit Bar-Ilan,et al.  Coverage and adoption of altmetrics sources in the bibliometric community , 2014, Scientometrics.

[8]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[9]  Lutz Bornmann,et al.  Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics , 2014, Scientometrics.

[10]  R. Merton.  The Matthew Effect in Science, II: Cumulative Advantage and the Symbolism of Intellectual Property , 1988, Isis.

[11]  Christian Pieter Hoffmann,et al.  Altmetrics for large, multidisciplinary research groups : Comparison of current tools , 2014 .

[12]  Michael Thelwall.  Webometrics and Altmetrics: Home Birth vs. Hospital Birth , 2016 .

[13]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[14]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[15]  Henk F. Moed,et al.  New developments in the use of citation analysis in research evaluation , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[16]  Rodrigo Costas,et al.  How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric.com APIs , 2014 .

[17]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[18]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[19]  Yin Leng Theng,et al.  Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media , 2016, Scientometrics.

[20]  Ian Rowlands,et al.  Social media use in the research workflow , 2011, Inf. Serv. Use.

[21]  유화자.  기독교 사역과 Leadership , 1997 .

[22]  Rodrigo Costas,et al.  Interpreting "altmetrics": viewing acts on social media through the lens of citation and social theories , 2015, ArXiv.

[23]  E. Garfield.  Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[24]  Bhaskar Mukherjee,et al.  And now for something completely different: the congruence of the Altmetric Attention Score’s structure between different article groups , 2017, Scientometrics.

[25]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[26]  Vincent Larivière,et al.  Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter , 2014, J. Assoc. Inf. Sci. Technol..

[27]  Ian Rowlands,et al.  Social media use in the research workflow , 2011 .

[28]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[29]  Evaristo Jiménez-Contreras,et al.  Can we use altmetrics at the institutional level? A case study analysing the coverage by research areas of four Spanish universities , 2016, ArXiv.

[30]  Jing Chen,et al.  (+)-Rutamarin as a Dual Inducer of Both GLUT4 Translocation and Expression Efficiently Ameliorates Glucose Homeostasis in Insulin-Resistant Mice , 2012, PloS one.