Web indicators for research evaluation. Part 2: Social media metrics

This literature review assesses indicators derived from social media sources, including both general and academic sites. Such indicators have been termed altmetrics, influmetrics, social media metrics, or a type of webometric, and have recently been commercialised by a number of companies and employed by some publishers and university administrators. The social media metrics analysed here derive mainly from Twitter , Facebook , Google+ , F1000 , Mendeley , ResearchGate , and Academia.edu . They have the apparent potential to deliver fast, free indicators of the wider societal impact of research, or of different types of academic impacts, complementing academic impact indicators from traditional citation indexes. Although it is unwise to employ them in formal evaluations with stakeholders, due to their susceptibility to gaming and lack of real evidence that they reflect wider research impacts, they are useful for formative evaluations and to investigate science itself. Mendeley reader counts are particularly promising.

[1]  Mike Thelwall,et al.  Why do papers have many Mendeley readers but few Scopus-indexed citations and vice versa? , 2017, J. Libr. Inf. Sci..

[2]  Mike Thelwall,et al.  Mendeley readership counts: An investigation of temporal and disciplinary differences , 2016, J. Assoc. Inf. Sci. Technol..

[3]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[4]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[5]  Mike Thelwall,et al.  Can Amazon.com reviews help to assess the wider impacts of books? , 2016, J. Assoc. Inf. Sci. Technol..

[6]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[7]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[8]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[9]  Lutz Bornmann,et al.  Which people use which scientific papers? An evaluation of data from F1000 and Mendeley , 2015, J. Informetrics.

[10]  Rodrigo Costas,et al.  Do Mendeley Readership Counts Help to Filter Highly Cited WoS Publications better than Average Citation Impact of Journals (JCS)? , 2015, ISSI.

[11]  Mike Thelwall,et al.  The influence of time and discipline on the magnitude of correlations between citation counts and quality scores , 2015, J. Informetrics.

[12]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[13]  Mike Thelwall,et al.  ResearchGate: Disseminating, communicating, and measuring Scholarship? , 2015, J. Assoc. Inf. Sci. Technol..

[14]  S. Haustein,et al.  Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns , 2015, PloS one.

[15]  Lutz Bornmann,et al.  Who Publishes, Reads, and Cites Papers? An Analysis of Country Information , 2015, ISSI.

[16]  Daqing He,et al.  User participation in an academic social networking service: A survey of open group users on Mendeley , 2014, J. Assoc. Inf. Sci. Technol..

[17]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[18]  Ehsan Mohammadi,et al.  Identifying the invisible impact of scholarly publications : a multi-disciplinary analysis using 'altmetrics' , 2015 .

[19]  Stefanie Haustein,et al.  Exploring data quality and retrieval strategies for mendeley reader counts , 2015 .

[20]  Sibele Fausto,et al.  Does the Global South Have Altmetrics? Analyzing a Brazilian LIS Journal , 2015, ISSI.

[21]  Vincent Larivière,et al.  Is There a Gender Gap in Social Media Metrics? , 2015, ISSI.

[22]  Ali Ghazinejad,et al.  Who Tweets about Science? , 2015, ISSI.

[23]  Liz Allen,et al.  Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact , 2014, PLoS biology.

[24]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[25]  Vincent Larivière,et al.  Tweets vs. Mendeley readers: How do these two social media metrics differ? , 2014, it Inf. Technol..

[26]  Mike Thelwall,et al.  Do highly cited researchers successfully use the social web? , 2014, Scientometrics.

[27]  Christian Pieter Hoffmann,et al.  Altmetrics for large, multidisciplinary research groups : Comparison of current tools , 2014 .

[28]  Richard Furuta,et al.  Do altmetrics follow the crowd or does the crowd follow altmetrics? , 2014, IEEE/ACM Joint Conference on Digital Libraries.

[29]  Vincent Larivière,et al.  Astrophysicists on Twitter: An in-depth analysis of tweeting and scientific publication behavior , 2014, Aslib J. Inf. Manag..

[30]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[31]  Nicolás Robinson-García,et al.  New data, new possibilities: Exploring the insides of Altmetric.com , 2014, ArXiv.

[32]  Lutz Bornmann,et al.  Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics , 2014, J. Informetrics.

[33]  C Matthew Hawkins,et al.  Social media in radiology: early trends in Twitter microblogging at radiology's largest international meeting. , 2014, Journal of the American College of Radiology : JACR.

[34]  Mike Thelwall,et al.  Academia.edu: Social network or Academic Network? , 2014, J. Assoc. Inf. Sci. Technol..

[35]  Christoph Trattner,et al.  Twitter in academic conferences: usage, networking and participation over time , 2014, HT.

[36]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[37]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[38]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[39]  Beata Stachowiak,et al.  The Presence of Polish Academics on Social Networking Websites for Academics, Using the Example of Employees of Nicolaus Copernicus University , 2014 .

[40]  Judit Bar-Ilan,et al.  Astrophysics publications on arXiv, Scopus and Mendeley: a case study , 2013, Scientometrics.

[41]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[42]  Jaime A. Teixeira da Silva,et al.  The need for post-publication peer review in plant science publishing , 2013, Front. Plant Sci..

[43]  Jaime A. Teixeira da Silva,et al.  The need for post-publication peer review in plant science publishing , 2013, Front. Plant Sci..

[44]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[45]  M. Fenner What Can Article-Level Metrics Do for You? , 2013, PLoS biology.

[46]  Adam Eyre-Walker,et al.  The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations , 2013, PLoS biology.

[47]  Stacy Konkiel,et al.  Tracking citations and altmetrics for research data: Challenges and opportunities , 2013 .

[48]  Rousseau Ronald,et al.  A multi-metric approach for research evaluation , 2013 .

[49]  Arbana Kadriu,et al.  Discovering value in academic social networks: A case study in ResearchGate , 2013, Proceedings of the ITI 2013 35th International Conference on Information Technology Interfaces.

[50]  Mike D Cadogan,et al.  The impact of social media on a major international emergency medicine conference , 2013, Emergency Medicine Journal.

[51]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[52]  Loet Leydesdorff,et al.  The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 , 2012, J. Informetrics.

[53]  Mike Thelwall,et al.  Tweeting Links to Academic Articles , 2013 .

[54]  Jens Peter Andersen,et al.  Altmetrics: An Alternate Perspective on Research Evaluation , 2013 .

[55]  Amanda J Lee,et al.  Increased Use of Twitter at a Medical Conference: A Report and a Review of the Educational Opportunities , 2012, Journal of medical Internet research.

[56]  Jane Hunter,et al.  Post-Publication Peer Review: Opening Up Scientific Conversation , 2012, Front. Comput. Neurosci..

[57]  Xianwen Wang,et al.  Tracing scientist’s research trends realtimely , 2012, Scientometrics.

[58]  D. Crotty Life after publication: Post-publication peer review , 2012 .

[59]  Mark Kats,et al.  Tweeting the Meeting: An In-Depth Analysis of Twitter Activity at Kidney Week 2011 , 2012, PloS one.

[60]  Jenny Fry,et al.  Measuring researchers’ use of scholarly information through social bookmarking data: A case study of BibSonomy , 2012, J. Inf. Sci..

[61]  Judit Bar-Ilan,et al.  Beyond citations: Scholars' visibility on the social Web , 2012, ArXiv.

[62]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[63]  Nirmali Chakraborty,et al.  Activities and Reasons for Using Social Networking Sites by Research Scholars in NEHU: A Study on Facebook and ResearchGate , 2012 .

[64]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[65]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.

[66]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[67]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[68]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[69]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[70]  Melissa Terras,et al.  Enabled backchannel: conference Twitter use by digital humanists , 2011, J. Documentation.

[71]  Massimo Franceschet,et al.  The first Italian research assessment exercise: A bibliometric perspective , 2009, J. Informetrics.

[72]  Katrin Weller,et al.  Citation Analysis in Twitter: Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences , 2011, #MSM.

[73]  R. Steen Retractions in the scientific literature: do authors deliberately commit research fraud? , 2010, Journal of Medical Ethics.

[74]  Jason Priem,et al.  How and why scholars cite on Twitter , 2010, ASIST.

[75]  Bradley M. Hemminger,et al.  Scientometrics 2.0: New metrics of scholarly impact on the social Web , 2010, First Monday.

[76]  D. Wardle Do 'Faculty of 1000' (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? , 2010 .

[77]  John G. Breslin,et al.  Using Twitter During an Academic Conference: The #iswc2009 Use-Case , 2010, ICWSM.

[78]  Chih-Ping Wei,et al.  Understanding what concerns consumers: a semantic approach to product feature extraction from consumer reviews , 2010, Inf. Syst. E Bus. Manag..

[79]  John G. Breslin,et al.  Understanding how Twitter is used to spread scientific messages , 2010 .

[80]  C. Neylon,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2009, PLoS biology.

[81]  Victor Henning,et al.  Mendeley - A Last.fm For Research? , 2008, 2008 IEEE Fourth International Conference on eScience.

[82]  Antal van den Bosch,et al.  Recommending scientific articles using citeulike , 2008, RecSys '08.

[83]  Steve Pettifer,et al.  Defrosting the Digital Library: Bibliographic Tools for the Next Generation Web , 2008, PLoS Comput. Biol..

[84]  Dario Taraborelli,et al.  Soft peer review. Social software and distributed scientific evaluation , 2008, COOP.

[85]  L. Butler,et al.  Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: an economics pilot study , 2007 .

[86]  Jon Ritterbush,et al.  Supporting Library Research with LibX and Zotero , 2007 .

[87]  Johan Bollen,et al.  Toward alternative metrics of journal impact: A comparison of download and citation data , 2005, Inf. Process. Manag..

[88]  Stephen S. Murray,et al.  The bibliometric properties of article readership information , 2005, J. Assoc. Inf. Sci. Technol..

[89]  Charles Oppenheim,et al.  The Influence of Peer Review on the Research Assessment Exercise , 2004, J. Inf. Sci..

[90]  Stéfan Jacques Darmoni,et al.  Reading factor as a credible alternative to impact factor: a preliminary study , 2000 .

[91]  Michael J. Kurtz,et al.  The NASA Astrophysics Data System: Overview , 2000, astro-ph/0002104.

[92]  B. Cronin,et al.  The praxis of acknowledgement: from bibliometrics to influmetrics , 1995 .