Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed.

[1]  Edward Nason,et al.  Health research: measuring the social, health and economic benefits , 2009, Canadian Medical Association Journal.

[2]  Sibele Fausto,et al.  Blogging : Indexing and Registering the Change in Science 2 . 0 , 2012 .

[3]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[4]  Jens Terliesner,et al.  Scientific bloggers under the altmetric microscope , 2012, ASIST.

[5]  Stephen J. Bensman.  The evaluation of research by scientometric indicators , 2011, J. Assoc. Inf. Sci. Technol..

[6]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[7]  Hui Wu,et al.  Correlation and Interaction Visualization of Altmetric Indicators Extracted From Scholarly Social Network Activities: Dimensions and Structure , 2013, Journal of medical Internet research.

[8]  Tara J Brigham.  An Introduction to Altmetrics , 2014, Medical reference services quarterly.

[9]  Mike Taylor,et al.  Exploring the Boundaries: How Altmetrics Can Expand Our Vision of Scholarly Communication and Social Impact , 2013 .

[10]  Stacy Konkiel,et al.  New opportunities for repositories in the age of altmetrics , 2013 .

[11]  Stacy Konkiel,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2016 .

[12]  Martin Fenner.  Letter from the Guest Content Editor: Altmetrics Have Come of Age , 2013 .

[13]  Nicolás Robinson-García,et al.  The Google scholar experiment: How to index false papers and manipulate bibliometric indicators , 2013, J. Assoc. Inf. Sci. Technol..

[14]  Lutz Bornmann,et al.  Is there currently a scientific revolution in Scientometrics? , 2013, J. Assoc. Inf. Sci. Technol..

[15]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[16]  Lutz Bornmann,et al.  How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects , 2013, J. Informetrics.

[17]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[18]  S. Schwartzman,et al.  The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies , 1994 .

[19]  Blaise Cronin,et al.  The evolving indicator space (iSpace) , 2013, J. Assoc. Inf. Sci. Technol..

[20]  Judit Bar-Ilan,et al.  Coverage and adoption of altmetrics sources in the bibliometric community , 2014, Scientometrics.

[21]  Anne E. Rauh,et al.  Introduction to Altmetrics for Science, Technology, Engineering, and Mathematics (STEM) Librarians , 2013 .

[22]  Holly M. Bik,et al.  An Introduction to Social Media for Scientists , 2013, PLoS biology.

[23]  Peter Vinkler,et al.  The Evaluation of Research by Scientometric Indicators , 2010 .

[24]  Loet Leydesdorff,et al.  The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits , 2012, J. Informetrics.

[25]  Bradley M. Hemminger,et al.  Scientometrics 2.0: New metrics of scholarly impact on the social Web , 2010, First Monday.

[26]  William Gunn,et al.  Social Signals Reflect Academic Impact: What it Means When a Scholar Adds a Paper to Mendeley , 2013 .

[27]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[28]  Patrick Dunleavy,et al.  The Impact of the Social Sciences: How Academics and their Research Make a Difference , 2014 .

[29]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[30]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[31]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[32]  Martin Fenner,et al.  Altmetrics in Evolution: Defining and Redefining the Ontology of Article-Level Metrics , 2013 .

[33]  Evaristo Jiménez-Contreras,et al.  Altmetrics: New Indicators for Scientific Communication in Web 2.0 , 2013, ArXiv.

[34]  Scott Chamberlain,et al.  Consuming Article-Level Metrics: Observations and Lessons , 2013 .

[35]  Sarah M. Glaser,et al.  Science Communication in a Digital Age: Social Media and the American Fisheries Society , 2013 .

[36]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[37]  A. Darzi,et al.  Twitter and the health reforms in the English National Health Service. , 2013, Health policy.

[38]  Katrin Weller,et al.  Citations in Web 2.0 , 2012 .

[39]  M. Fenner.  What Can Article-Level Metrics Do for You? , 2013, PLoS biology.

[40]  Jean Liu Data Curator.  Five challenges in altmetrics: A toolmaker's perspective , 2013 .

[41]  Mercer Jennifer Ann,et al.  PUBLICATION manual of the American Psychological Association. , 1952, Psychological bulletin.

[42]  Lutz Bornmann,et al.  What is societal impact of research and how can it be assessed? a literature survey , 2013, J. Assoc. Inf. Sci. Technol..

[43]  M. Thelwall,et al.  Google Scholar citations and Google Web-URL citations: A multi-discipline exploratory analysis , 2007 .

[44]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[45]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[46]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[47]  Lutz Bornmann,et al.  Measuring the societal impact of research: references to climate change research in relevant policy literature , 2016 .

[48]  Emily S. Darling,et al.  The role of twitter in the life cycle of a scientific publication , 2013, ArXiv.

[49]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[50]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[51]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[52]  Cornelius Puschmann,et al.  (Micro)Blogging Science? Notes on Potentials and Constraints of New Forms of Scholarly Communication , 2014 .

[53]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[54]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[55]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[56]  Cassidy R. Sugimoto,et al.  Bibliographic References in Web 2.0 , 2014 .

[57]  Heather A. Piwowar,et al.  Altmetrics: Value all research products , 2013, Nature.

[58]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[59]  Christine Greenhow,et al.  Social scholarship: Reconsidering scholarly practices in the age of social media , 2014, Br. J. Educ. Technol..

[60]  Johan Bollen,et al.  Usage bibliometrics , 2011, Annu. Rev. Inf. Sci. Technol..

[61]  Sarah Barbrow,et al.  A Look at Altmetrics and Its Growing Significance to Research Libraries , 2013 .

[62]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[63]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[64]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[65]  Heather A. Piwowar,et al.  The power of altmetrics on a CV , 2013 .

[66]  R. Roemer,et al.  From bibliometrics to altmetrics A changing scholarly landscape , 2012 .

[67]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[68]  Ronald Rousseau,et al.  A Multi-metric Approach for Research Evaluations , 2022 .

[69]  Mike Thelwall,et al.  How is research blogged? A content analysis approach , 2015, J. Assoc. Inf. Sci. Technol..

[70]  Katrin Weller,et al.  Citation Analysis in Twitter: Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences , 2011, #MSM.

[71]  Robin Chin Roemer,et al.  Institutional Altmetrics and Academic Libraries , 2013 .

[72]  Cameron Neylon,et al.  Rethinking Impact: Applying Altmetrics to Southern African Research , 2014 .

[73]  Isabella Peters,et al.  Using social bookmarks and tags as alternative indicators of journal content description , 2012, First Monday.

[74]  Daniel Torres-Salinas,et al.  Altmetrics: nuevos indicadores para la comunicación científica en la Web 2.0 , 2013 .