Research on the impact evaluation of academic journals based on altmetrics and citation indicators

Taking international LIS journals as an example, combining 8 traditional citation indicators and 10 altmetrics indicators, we use the method of multi‐indicator fusion to construct a comprehensive evaluation model of journal impact based on traditional citation indicators and altmetrics indicators. Through normalization processing, correlation analysis, reliability analysis, validity analysis, factor analysis, the paper evaluates the academic impact (FA), societal impact (FS) of LIS journals respectively. We get the following conclusion: (1) We get the 2‐dimensional evaluation results and according to LIS journals impacts, we divide the roles into four categories: “Prestige journals”, “Star journals”, “Common journals” and “Expert journals”. (2) It is found that 8 traditional citation indicators based on JCR, present strong positive correlations, and the indicators are highly consistent overall and internally. (3) There are moderate or high positive correlations among 10 altmetrics based on Altmetrics Explorer, and there are also significant consistency among the indicators as a whole and internally. (4) The correlation coefficient between FA and FS is 0.566, with a moderate positive correlation. It indicates that the evaluation of LIS journals' societal impact based on altmetrics indicators has a useful supplement to the evaluation of academic impact based on citation.

[1]  Lutz Bornmann,et al.  How to normalize Twitter counts? A first attempt based on journals in the Twitter Index , 2016, Scientometrics.

[2]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[3]  Mike Thelwall,et al.  How quickly do publications get read? The evolution of mendeley reader counts for new articles , 2018, J. Assoc. Inf. Sci. Technol..

[4]  Mike Thelwall,et al.  Are Mendeley reader counts high enough for research evaluations when articles are published? , 2017, Aslib J. Inf. Manag..

[5]  Houqiang Yu,et al.  Context of altmetrics data matters: an investigation of count type and user category , 2017, Scientometrics.

[6]  Henk F. Moed,et al.  Multidimensional assessment of scholarly research impact , 2014, J. Assoc. Inf. Sci. Technol..

[7]  Katherine G. Akers.  Introducing altmetrics to the Journal of the Medical Library Association , 2017, Journal of the Medical Library Association : JMLA.

[8]  Lutz Bornmann,et al.  Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics , 2014, J. Informetrics.

[9]  R. Merton.  The Matthew Effect in Science, II: Cumulative Advantage and the Symbolism of Intellectual Property , 1988, Isis.

[10]  Kai Li,et al.  Web of Science use in published research and review papers 1997–2017: a selective, dynamic, cross-domain, content-based analysis , 2017, Scientometrics.

[11]  Vincent Larivière,et al.  Scholarly use of social media and altmetrics: A review of the literature , 2016, J. Assoc. Inf. Sci. Technol..

[12]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[13]  José Luis Ortega,et al.  The presence of academic journals on Twitter and its relationship with dissemination (tweets) and research impact (citations) , 2017, Aslib J. Inf. Manag..

[14]  Dietmar Wolfram,et al.  Difference in the impact of open-access papers published by China and the USA , 2018, Scientometrics.

[15]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[16]  E. Garfield.  Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[17]  P. T. P. OLIVER,et al.  Citation Indexing for Studying Science , 1970, Nature.

[18]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[19]  Wolfgang Glänzel,et al.  Usage metrics versus altmetrics: confusing terminology? , 2015, Scientometrics.

[20]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1975, Journal for the Scientific Study of Religion.

[21]  Mike Thelwall,et al.  Could scientists use Altmetric.com scores to predict longer term citation counts? , 2018, J. Informetrics.

[22]  R. Werner.  The focus on bibliometrics makes papers less useful , 2015, Nature.

[23]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[24]  Liwei Zhang,et al.  Why highly cited articles are not highly tweeted? A biology case , 2018, Scientometrics.