Investigating familiarity and usage of traditional metrics and altmetrics

As the online dissemination of scholarly outputs gets faster and easier, altmetrics, social media based indices, have emerged alongside traditional metrics for research evaluation. In a two‐phase survey, we investigate scholars' familiarity and usage of traditional metrics and altmetrics. In this paper, we present the second phase with 448 participants. We found few traditional metrics, like the Journal Impact Factor and number of citations, are familiar to and often used by scholars for research evaluation. Among altmetrics, only views/downloads, readers, and followers are known to more than half the respondents. Unseen benefits and lack of time are hindrances to using metrics for the evaluation of research outputs. Although social media are well‐known, scholars prefer promoting their research by publishing in journals and attending conferences. We found social media usage, perceived ease of use and usefulness of altmetrics affect the usage of altmetrics. Findings suggest altmetrics have attracted attention in academia and could be considered complementary to traditional metrics. We acknowledge that due to the limited sample size, statistics and demographics in this study, findings cannot be said to be representative of the entire academic population worldwide. Future studies are needed that cover a wider range of academic disciplines around the world.

[1]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[2]  Robert G. Capra,et al.  Factors mediating disclosure in social network sites , 2011, Comput. Hum. Behav..

[3]  G. Kirkup Academic blogging: academic practice and academic identity , 2010 .

[4]  Vassilis Moustakis,et al.  Modeling the acceptance of clinical information systems among hospital medical staff: An extended TAM model , 2011, J. Biomed. Informatics.

[5]  L. Bonetta Scientists Enter the Blogosphere , 2007, Cell.

[6]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[7]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[8]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[9]  Wynne W. Chin,et al.  On the use, usefulness, and ease of use of structural equation modeling in MIS research: a note of caution , 1995 .

[10]  R. Merton The Matthew Effect in Science , 1968, Science.

[11]  Lucy Dipper,et al.  The effect of social media promotion on academic article uptake , 2017, J. Assoc. Inf. Sci. Technol..

[12]  Vincent Larivière,et al.  Scholarly use of social media and altmetrics: A review of the literature , 2016, J. Assoc. Inf. Sci. Technol..

[13]  Henk F. Moed,et al.  Journal impact measures in bibliometric research , 2004, Scientometrics.

[14]  Jean Liu Data Curator Five challenges in altmetrics: A toolmaker's perspective , 2013 .

[15]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[16]  R. Procter,et al.  Adoption and use of Web 2.0 in scholarly communications , 2010, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[17]  Stefanie Haustein,et al.  Grand challenges in altmetrics: heterogeneity, data quality and dependencies , 2016, Scientometrics.

[18]  Katrin Weller Social Media and Altmetrics: An Overview of Current Alternative Approaches to Measuring Scholarly Impact , 2015 .

[19]  Emma Weitkamp,et al.  A Case Study in Serendipity: Environmental Researchers Use of Traditional and Social Media for Dissemination , 2013, PloS one.

[20]  H. Moed,et al.  The use of bibliometric data for the measurement of university research performance , 1985 .

[21]  Michael R. Mullen,et al.  Structural equation modelling: guidelines for determining model fit , 2008 .

[22]  Jonathan Furner,et al.  Scholarly communication and bibliometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[23]  Magid Igbaria,et al.  Personal Computing Acceptance Factors in Small Firms: A Structural Equation Model , 1997, MIS Q..

[24]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[25]  Judit Bar-Ilan,et al.  Coverage and adoption of altmetrics sources in the bibliometric community , 2014, Scientometrics.

[26]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[27]  Wolfgang Glänzel,et al.  Usage metrics versus altmetrics: confusing terminology? , 2015, Scientometrics.

[28]  Anatoliy A. Gruzd,et al.  Tenure and promotion in the age of online social media , 2011, ASIST.

[29]  Yin Leng Theng,et al.  Awareness and usage of altmetrics: A user survey , 2017, ASIST.

[30]  Eric Schoop,et al.  What we blog? A qualitaive analysis of researchers' weblogs , 2010, ELPUB.

[31]  Executive Summary,et al.  Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines - Executive Summary , 2010 .

[32]  Ellen Collins,et al.  Use and relevance of Web 2.0 resources for researchers , 2010, ELPUB.

[33]  Jean Liu,et al.  Five challenges in altmetrics: A toolmaker's perspective , 2013 .

[34]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[35]  Carl T. Bergstrom,et al.  The Eigenfactor™ Metrics , 2008, The Journal of Neuroscience.

[36]  Tim S. Evans,et al.  Ranking Journals Using Altmetrics , 2015, ISSI.

[37]  Geoffrey S. Hubona,et al.  The mediation of external variables in the technology acceptance model , 2006, Inf. Manag..

[38]  Daipayan Guha,et al.  Social Media Metrics and Bibliometric Profiles of Neurosurgical Departments and Journals: Is There a Relationship? , 2016, World neurosurgery.

[39]  S. Hansson,et al.  Impact factor as a misleading tool in evaluation of medical journals , 1995, The Lancet.

[40]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[41]  María José Luzón,et al.  Scholarly hyperwriting: The function of links in academic weblogs , 2009, J. Assoc. Inf. Sci. Technol..

[42]  Steven Ovadia,et al.  When Social Media Meets Scholarly Publishing , 2013 .

[43]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[44]  David Stuart,et al.  Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact , 2015, Online information review (Print).

[45]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[46]  R. Kline Promise and pitfalls of structural equation modeling in gifted research. , 2010 .

[47]  Anatoliy A. Gruzd,et al.  Connected scholars: Examining the role of social media in research practices of faculty using the UTAUT model , 2012, Comput. Hum. Behav..

[48]  David Stuart Social Media Metrics , 2009 .

[49]  Yin Leng Theng,et al.  Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media , 2016, Scientometrics.

[50]  Ian Rowlands,et al.  Social media use in the research workflow , 2011, Inf. Serv. Use.

[51]  Isabel Gómez,et al.  Advantages and limitations in the use of impact factor measures for the assessment of research performance , 2002, Scientometrics.

[52]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[53]  C. Strasser,et al.  Researcher Perspectives on Publication and Peer Review of Data , 2014, PloS one.

[54]  Heather A. Piwowar,et al.  Riding the crest of the altmetrics wave: How librarians can help prepare faculty for the next generation of research impact metrics , 2013, College & Research Libraries News.

[55]  GorraizJuan,et al.  Usage metrics versus altmetrics , 2015 .

[56]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[57]  Gunilla Widén,et al.  Scholarly communication and possible changes in the context of social media: A Finnish case study , 2011, Electron. Libr..

[58]  A A Sandberg,et al.  The journal "impact factor": a misnamed, misleading, misused measure. , 1998, Cancer genetics and cytogenetics.

[59]  Diana Cardona-Grau,et al.  Introducing the Twitter Impact Factor: An Objective Measure of Urology's Academic Impact on Twitter. , 2016, European urology focus.

[60]  Ming-Yueh Tsay 學術傳播與書目計量學 Scholarly Communication and Bibliometrics , 1997 .

[61]  C. Stein,et al.  Structural equation modeling. , 2012, Methods in molecular biology.

[62]  Blaise Cronin,et al.  The citation process: The role and significance of citations in scientific communication , 1984 .

[63]  L. Naylor,et al.  Assessing Awareness, Knowledge and Use of Folic Acid in Kansas Women between the Ages of 18 and 44 Years , 2008 .