Issues in the Interpretation of “Altmetrics” Digital Traces: A Review

Researchers leave traces of their behavior during many stages of their research process. Parts of this process were formerly invisible. With scholarship moving online, we can now access various types of altmetrics digital traces such as reading, organizing, sharing and discussing scientific papers, thus develop a more holistic story about researchers and their work. However, a lack of in-depth interpretation of altmetrics digital traces is observed. Therefore, this paper focuses on reviewing some of existing altmetrics research, with a particular emphasis on the issues that need to be taken into consideration in the interpretation of altmetrics digital traces. Taking a preliminary step towards a guideline for more in-depth analysis of digital traces of scholarly acts, this review aims to bring attention to these issues to avoid misuse of altmetrics indicators.

[1]  William Gunn,et al.  Social Signals Reflect Academic Impact: What it Means When a Scholar Adds a Paper to Mendeley , 2013 .

[2]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[3]  Sushil Jajodia,et al.  Detecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg? , 2012, IEEE Transactions on Dependable and Secure Computing.

[4]  Loet Leydesdorff,et al.  The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits , 2012, J. Informetrics.

[5]  Rosanna E. Guadagno,et al.  Gender differences in mediated communication: Women connect more than do men , 2013, Comput. Hum. Behav..

[6]  Cassidy R. Sugimoto,et al.  Science Metrics and Science Policy , 2014 .

[7]  C. Neylon,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2009, PLoS biology.

[8]  Shenmeng Xu,et al.  What flavors are different types of scholarly articles? An investigation of PLOS publications , 2015, ASIST.

[9]  Michael J. Moravcsik,et al.  Variation of the nature of citation measures with journals and scientific specialties , 1978, J. Am. Soc. Inf. Sci..

[10]  Thorsten Gruber.  Academic sell-out: how an obsession with metrics and rankings is damaging academia , 2014 .

[11]  Wang Mei-ying,et al.  Correlation between Download and Citation and Download-citation Deviation Phenomenon for Some Papers in Chinese Medical Journals , 2011 .

[12]  Shenmeng Xu,et al.  Global science discussed in local altmetrics: Weibo and its comparison with Twitter , 2017, J. Informetrics.

[13]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[14]  Mike Thelwall.  Journal impact evaluation: a webometric perspective , 2012, Scientometrics.

[15]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[16]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[17]  Ali Ghazinejad,et al.  Who Tweets about Science? , 2015, ISSI.

[18]  Jenny Fry,et al.  Measuring researchers’ use of scholarly information through social bookmarking data: A case study of BibSonomy , 2012, J. Inf. Sci..

[19]  S E Cozzens.  The character of science: changing order. , 1985, Science.

[20]  Ullica Segerstrale,et al.  Beyond the Science Wars: The Missing Discourse about Science and Society , 2001 .

[21]  Juan Gorraiz,et al.  Global usage versus global citation metrics: The case of pharmacology journals , 2011, J. Assoc. Inf. Sci. Technol..

[22]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[23]  Henry Small.  Referencing as Cooperation or Competition , 2016 .

[24]  Vincent Larivière,et al.  Scholarly use of social media and altmetrics: A review of the literature , 2016, J. Assoc. Inf. Sci. Technol..

[25]  Martin Fenner,et al.  Altmetrics in Evolution: Defining and Redefining the Ontology of Article-Level Metrics , 2013 .

[26]  M. HamidR.Jamali,et al.  User diversity: as demonstrated by deep log analysis , 2008, Electron. Libr..

[27]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[28]  Henk F. Moed,et al.  Statistical relationships between downloads and citations at the level of individual documents within a single journal , 2005, J. Assoc. Inf. Sci. Technol..

[29]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[30]  David Crotty.  Altmetrics: Finding Meaningful Needles in the Data Haystack , 2014 .

[31]  Finn Årup Nielsen,et al.  Scientific citations in Wikipedia , 2007, First Monday.

[32]  F. Shapiro,et al.  The Yale book of quotations , 2006 .

[33]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[34]  Sadie Creese,et al.  Applying Social Network Analysis to Security , 2015 .

[35]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[36]  S. Haustein,et al.  Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns , 2015, PloS one.

[37]  H. Moed,et al.  The use of bibliometric data for the measurement of university research performance , 1985 .

[38]  Karin Knorr Cetina,et al.  Epistemic Cultures: Forms of Reason in Science , 1991 .

[39]  Johan Bollen,et al.  Usage bibliometrics , 2011, Annu. Rev. Inf. Sci. Technol..

[40]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[41]  Rodrigo Costas,et al.  Interpreting "altmetrics": viewing acts on social media through the lens of citation and social theories , 2015, ArXiv.

[42]  Stevan Harnad,et al.  Earlier Web Usage Statistics as Predictors of Later Citation Impact , 2005, J. Assoc. Inf. Sci. Technol..

[43]  Gohar Feroz Khan,et al.  Editorial: Triple Helix and innovation in Asia using scientometrics, webometrics, and informetrics , 2011, Scientometrics.

[44]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[45]  Paul Groth,et al.  Studying Scientific Discourse on the Web Using Bibliometrics: A Chemistry Blogging Case Study , 2010 .

[46]  Lutz Bornmann,et al.  Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics , 2014, J. Informetrics.

[47]  Euan Adie,et al.  Taking the alternative mainstream , 2014 .

[48]  Mike Thelwall,et al.  Bibliometrics to webometrics , 2008, J. Inf. Sci..

[49]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[50]  Stacy Konkiel.  Grand challenges in altmetrics: heterogeneity, data quality and dependencies , 2017 .

[51]  C. Bazerman.  Changing Order: Replication and Induction in Scientific Practice , 1989 .

[52]  Emily S. Darling,et al.  The role of twitter in the life cycle of a scientific publication , 2013, ArXiv.

[53]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[54]  Shenmeng Xu,et al.  Who, what, why? An exploration of JoVE scientific video publications in tweets , 2018, Scientometrics.

[55]  Peter Ingwersen,et al.  Toward a basic framework for webometrics , 2004, J. Assoc. Inf. Sci. Technol..

[56]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[57]  Liwen Vaughan,et al.  Webometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[58]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[59]  Quentin L. Burrell,et al.  Predicting future citation behavior , 2003, J. Assoc. Inf. Sci. Technol..

[60]  Jin-Cheon Na,et al.  Content analysis of scholarly discussions of psychological academic articles on Facebook , 2017, Online Inf. Rev..

[61]  Ronald Rousseau,et al.  The journal download immediacy index (DII): experiences using a Chinese full-text database , 2010, Scientometrics.

[62]  Stephen Cole,et al.  Social Stratification in Science , 1974 .

[63]  B. Cronin,et al.  Criteria for Evaluating Indicators , 2014 .

[64]  Kim Holmberg,et al.  Highly tweeted science articles: who tweets them? An analysis of Twitter user profile descriptions , 2017, Scientometrics.

[65]  U. Oberst.  Measuring the Societal Impact of Research with Altmetrics: An Experiment , 2017 .

[66]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[67]  V. Cano,et al.  Citation life cycles of ten citation classics , 1991, Scientometrics.

[68]  C. D. Hurt.  Conceptual citation differences in science, technology, and social sciences literature , 1987, Inf. Process. Manag..

[69]  Tim Brody,et al.  Earlier Web usage statistics as predictors of later citation impact: Research Articles , 2006 .

[70]  Johan Bollen,et al.  Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics , 2006, J. Assoc. Inf. Sci. Technol..

[71]  Julia Lane,et al.  Science Metrics and Science Policy , 2013 .

[72]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..