Validating online reference managers for scholarly impact measurement

This paper investigates whether CiteULike and Mendeley are useful for measuring scholarly influence, using a sample of 1,613 papers published in Nature and Science in 2007. Traditional citation counts from the Web of Science (WoS) were used as benchmarks to compare with the number of users who bookmarked the articles in one of the two free online reference manager sites. Statistically significant correlations were found between the user counts and the corresponding WoS citation counts, suggesting that this type of influence is related in some way to traditional citation-based scholarly impact but the number of users of these systems seems to be still too small for them to challenge traditional citation indexes.

[1]  松田 直人.  『Google Scholar』の利点 , 2009 .

[2]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[3]  KoushaKayvan,et al.  Google Scholar citations and Google Web-URL citations: A multi-discipline exploratory analysis , 2007 .

[4]  Richard E. West,et al.  Mendeley: Creating Communities of Scholarly Inquiry Through Research Collaboration , 2011 .

[5]  Rob Procter,et al.  If you build it, will they come? : how researchers perceive and use web 2.0 : a Research Information Network report , 2010 .

[6]  Alastair G. Smith.  Wikipedia and institutional repositories: an academic symbiosis? , 2011 .

[7]  Charles Oppenheim,et al.  Peer review and the h-index: Two studies , 2010, J. Informetrics.

[8]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008, J. Assoc. Inf. Sci. Technol..

[9]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007 .

[10]  Stevan Harnad,et al.  Earlier Web Usage Statistics as Predictors of Later Citation Impact , 2005, J. Assoc. Inf. Sci. Technol..

[11]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[12]  Douglas K. Barry.  Chapter 8 – Change Will Happen , 2003 .

[13]  Isidro F. Aguillo.  Is Google Scholar useful for bibliometrics? A webometric analysis , 2012, Scientometrics.

[14]  Mike Thelwall,et al.  Bibliometrics to webometrics , 2008, J. Inf. Sci..

[15]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[16]  Liwen Vaughan,et al.  Web citation data for impact assessment: A comparison of four science disciplines: Book Reviews , 2005 .

[17]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[18]  David Osimo,et al.  Science 2.0 (change will happen....) , 2010, First Monday.

[19]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[20]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009, J. Assoc. Inf. Sci. Technol..

[21]  Richard K. Belew,et al.  Scientific impact quantity and quality: Analysis of two sources of bibliographic data , 2005, ArXiv.

[22]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[23]  Mike Thelwall,et al.  Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis , 2007, J. Assoc. Inf. Sci. Technol..

[24]  Paul Groth,et al.  Studying Scientific Discourse on the Web Using Bibliometrics: A Chemistry Blogging Case Study , 2010 .

[25]  Mark Ware,et al.  Peer review in scholarly journals: Perspective of the scholarly community - Results from an international study , 2008, Inf. Serv. Use.

[26]  Katrin Weller,et al.  Twitter for Scientific Communication: How Can Citations/References be Identified and Measured? , 2011 .

[27]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009 .

[28]  Rob Procter,et al.  If you build it, will they come? How researchers perceive and use web 2.0 , 2010 .

[29]  Mike Thelwall,et al.  Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals , 2008, J. Assoc. Inf. Sci. Technol..

[30]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..

[31]  Victor Henning,et al.  Mendeley - A Last.fm For Research? , 2008, 2008 IEEE Fourth International Conference on eScience.

[32]  Bradley M. Hemminger,et al.  Scientometrics 2.0: New metrics of scholarly impact on the social Web , 2010, First Monday.

[33]  Tim Brody,et al.  Earlier Web usage statistics as predictors of later citation impact: Research Articles , 2006 .

[34]  Dario Taraborelli,et al.  Soft peer review. Social software and distributed scientific evaluation , 2008, COOP.

[35]  Mike Thelwall,et al.  Using the Web for research evaluation: The Integrated Online Impact indicator , 2010, J. Informetrics.

[36]  Daqing He,et al.  Social reference: aggregating online usage of scientific literature in CiteULike for clustering academic resources , 2011, JCDL '11.

[37]  C. Neylon,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2009, PLoS biology.

[38]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[39]  Jason Priem,et al.  How and why scholars cite on Twitter , 2010, ASIST.

[40]  Blaise Cronin,et al.  Bibliometrics and beyond: some thoughts on web-based citation analysis , 2001, J. Inf. Sci..