Evaluating altmetrics

The rise of the social web and its uptake by scholars has led to the creation of altmetrics, which are social web metrics for academic publications. These new metrics can, in theory, be used in an evaluative role, to give early estimates of the impact of publications or to give estimates of non-traditional types of impact. They can also be used as an information seeking aid: to help draw a digital library user’s attention to papers that have attracted social web mentions. If altmetrics are to be trusted then they must be evaluated to see if the claims made about them are reasonable. Drawing upon previous citation analysis debates and web citation analysis research, this article discusses altmetric evaluation strategies, including correlation tests, content analyses, interviews and pragmatic analyses. It recommends that a range of methods are needed for altmetric evaluations, that the methods should focus on identifying the relative strengths of influences on altmetric creation, and that such evaluations should be prioritised in a logical order.

[1]  M. Hakel,et al.  An Examination of Sources of Peer-Review Bias , 2006, Psychological science.

[2]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[3]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009 .

[4]  S. Ceci,et al.  Understanding current causes of women's underrepresentation in science , 2011, Proceedings of the National Academy of Sciences.

[5]  Jason Priem,et al.  How and why scholars cite on Twitter , 2010, ASIST.

[6]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[7]  Liwen Vaughan,et al.  Relationship between links to journal Web sites and impact factors , 2002, Aslib Proc..

[8]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[9]  P. Seglen,et al.  Citation rates and journal impact factors are not suitable for evaluation of research. , 1998, Acta orthopaedica Scandinavica.

[10]  T. Brooks.  Evidence of complex citer motivations , 1986, J. Am. Soc. Inf. Sci..

[11]  Lutz Bornmann,et al.  Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication , 2008, Scientometrics.

[12]  Mike Thelwall,et al.  A combined bibliometric indicator to predict article impact , 2011, Inf. Process. Manag..

[13]  Yu Xie,et al.  A Theoretical and Empirical Investigation , 2006 .

[14]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008 .

[15]  J. Ravetz.  Sociology of Science , 1972, Nature.

[16]  B. Cronin,et al.  The web of knowledge: a festschrift in honor of Eugene Garfield , 2000 .

[17]  Kristina Lerman,et al.  Pragmatic evaluation of folksonomies , 2011, WWW.

[18]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[19]  M. Thelwall,et al.  Google Scholar citations and Google Web-URL citations: A multi-discipline exploratory analysis , 2007 .

[20]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[21]  Mark Kats,et al.  Tweeting the Meeting: An In-Depth Analysis of Twitter Activity at Kidney Week 2011 , 2012, PloS one.

[22]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[23]  D. Cases,et al.  How can we investigate citation behavior?: a study of reasons for citing literature in communication , 2000 .

[24]  Adam Marcus,et al.  Science publishing: The paper is not sacred , 2011, Nature.

[25]  R. Whitley.  The Intellectual and Social Organization of the Sciences (Second Edition: with new introductory chapter entitled 'Science Transformed? The Changing Nature of Knowledge Production at the End of the Twentieth Century') , 1984 .

[26]  Donald Owen Case,et al.  How can we investigate citation behavior? A study of reasons for citing literature in communication , 2000, J. Am. Soc. Inf. Sci..

[27]  D F Horrobin,et al.  The philosophical basis of peer review and the suppression of innovation. , 1990, JAMA.

[28]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[29]  M. Mahoney.  Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[30]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[31]  Judit Bar-Ilan,et al.  Beyond citations: Scholars' visibility on the social Web , 2012, ArXiv.

[32]  R. Procter,et al.  Adoption and use of Web 2.0 in scholarly communications , 2010, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[33]  Liwen Vaughan,et al.  Web citation data for impact assessment: A comparison of four science disciplines: Book Reviews , 2005 .

[34]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1992, Scientometrics.

[35]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[36]  Mike Thelwall,et al.  Using the Web for research evaluation: The Integrated Online Impact indicator , 2010, J. Informetrics.

[37]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[38]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[39]  A. F. J. VAN RAAN,et al.  In matters of quantitative studies of science the fault of theorists is offering too little and asking too much , 1998, Scientometrics.

[40]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[41]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[42]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[43]  Mike Thelwall,et al.  Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals , 2008, J. Assoc. Inf. Sci. Technol..

[44]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[45]  Katrin Weller,et al.  Social Software in Academia: Three Studies on Users' Acceptance of Web 2.0 Services , 2010 .