Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An altmetric study of economics

Altmetrics are often praised as an alternative or complement to classic bibliometric metrics, especially in the social sciences discipline. However, empirical investigations of altmetrics concerning the social sciences are scarce. This study investigates the extent to which economic research is shared on social media platforms with an emphasis on mentions in policy documents in addition to other mentions such as Twitter or Facebook. Moreover, this study explores machine learning models to predict the likelihood of a research article being classified into the top-quality tier of a journal ranking based on the altmetric mentions. The included journal rankings are the academic journal guide (AJG), source normalized impact per paper (SNIP) and journal citation reports (JCR). The investigated journals have been selected based on the AJG list and extracted from Altmetric.com data. After applying extensive data cleaning on the extracted data, a final set of 55,560 journal article records is obtained. The results indicate that the average number of policy mentions of the publications of economics journals is higher than the other subject areas included in the AJG list. Moreover, the publications in top-ranking economic journals are more likely to have a higher average number of policy mentions. Policy and Twitter mentions are presented as the most significant and informative social media mentions in demonstrating the broader impact and dissemination of Economics discipline followed by Blogs, Facebook, Wikipedia, and News. The results show that Support Vector Machine and Logistic Regression performed best in classifying the journal ranking tiers i.e. SNIP-based with 77% accuracy, JCR-based with 71% accuracy, and AJG-based with 66% accuracy. The models classified the ranking tier AJG18 with lower accuracy than SNIP and JCR. This might be because the AJG18 rankings are based on expert opinion, whereas SNIP and JCR are based on citations.

[1]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.

[2]  Andreas Thor,et al.  Which early works are cited most frequently in climate change research literature? A bibliometric approach based on Reference Publication Year Spectroscopy , 2016, Scientometrics.

[3]  Saeed-Ul Hassan,et al.  Deep context of citations using machine-learning models in scholarly full-text articles , 2018, Scientometrics.

[4]  Michael E. Papka,et al.  Exploring Features for Predicting Policy Citations , 2017, 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL).

[5]  Christopher Carroll,et al.  What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield , 2018, Front. Res. Metr. Anal..

[6]  Elizabeth Joan Kelly,et al.  Altmetrics and Archives , 2017 .

[7]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[8]  Kristin L. Sainani,et al.  Logistic Regression , 2014, PM & R : the journal of injury, function, and rehabilitation.

[9]  Sei-Ching Joanna Sin,et al.  Investigating familiarity and usage of traditional metrics and altmetrics , 2019, J. Assoc. Inf. Sci. Technol..

[10]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[11]  Lutz Bornmann,et al.  How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data , 2016, Scientometrics.

[12]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[13]  Lutz Bornmann,et al.  Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents? , 2015, Scientometrics.

[14]  Francisco Herrera,et al.  Predicting literature's early impact with sentiment analysis in Twitter , 2020, Knowl. Based Syst..

[15]  Thed N. van Leeuwen,et al.  Some modifications to the SNIP journal impact indicator , 2012, J. Informetrics.

[16]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[17]  Rabeeh Ayaz Abbasi,et al.  Mining network-level properties of Twitter altmetrics data , 2019, Scientometrics.

[18]  Saeed-Ul Hassan,et al.  Altmetrics Study of Economics , 2019, ISSI.

[19]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[20]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[21]  Péter Jacsó,et al.  Journal Citation Reports.">Five-year impact factor data in the Journal Citation Reports , 2009, Online Inf. Rev..

[22]  Kaltrina Nuredini,et al.  Enriching the knowledge of altmetrics studies by exploring social media metrics for Economic and Business Studies journals , 2016 .

[23]  Mark R. Segal,et al.  Machine Learning Benchmarks and Random Forest Regression , 2004 .

[24]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[25]  Francisco Herrera,et al.  Tweet Coupling: a social media methodology for clustering scientific publications , 2020, Scientometrics.

[26]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[27]  Kaltrina Nuredini,et al.  Economic and Business Studies Journals and Readership Information from Mendeley , 2015, ISI.

[28]  Fereshteh Didegah,et al.  Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data , 2017, Scientometrics.

[29]  Rodrigo Costas,et al.  How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric.com APIs , 2014 .

[30]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[31]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[32]  Saeed-Ul Hassan,et al.  Influential tweeters in relation to highly cited articles in altmetric big data , 2019, Scientometrics.

[33]  Salem Alelyani,et al.  Introducing the ‘alt-index’ for measuring the social visibility of scientific research , 2020, Scientometrics.