Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science

This study examines the differences between Scopus and Web of Science in the citation counting, citation ranking, and h-index of 22 top human-computer interaction (HCI) researchers from EQUATOR—a large British Interdisciplinary Research Collaboration project. Results indicate that Scopus provides significantly more coverage of HCI literature than Web of Science, primarily due to coverage of relevant ACM and IEEE peer-reviewed conference proceedings. No significant differences exist between the two databases if citations in journals only are compared. Although broader coverage of the literature does not significantly alter the relative citation ranking of individual researchers, Scopus helps distinguish between the researchers in a more nuanced fashion than Web of Science in both citation counting and h-index. Scopus also generates significantly different maps of citation networks of individual scholars than those generated by Web of Science. The study also presents a comparison of h-index scores based on Google Scholar with those based on the union of Scopus and Web of Science. The study concludes that Scopus can be used as a sole data source for citation-based research and evaluation in HCI, especially when citations in conference proceedings are sought, and that researchers should manually calculate h scores instead of relying on system calculations. © 2008 Wiley Periodicals, Inc.

[1]  Marcia Henry,et al.  Citation searching : New players, new tools , 2006 .

[2]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[3]  Lei Wang,et al.  Three options for citation tracking: Google Scholar, Scopus and Web of Science , 2006, Biomedical digital libraries.

[4]  Lokman I. Meho.  The Rise and Rise of Citation Analysis , 2006, ArXiv.

[5]  Henk F. Moed,et al.  UK Research Assessment Exercises: Informed judgments on research quality or quantity? , 2008, Scientometrics.

[6]  Lokman I. Meho,et al.  Using the h-index to rank influential information scientists , 2006, J. Assoc. Inf. Sci. Technol..

[7]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[8]  L. Butler,et al.  Assessing university research: A plea for a balanced approach , 2007 .

[9]  Oscar Volij,et al.  The Measurement of Intellectual Influence , 2002 .

[10]  Daniel S. Weld.  Comparative Analysis , 1987, IJCAI.

[11]  P. Jacsó.  As we may search : Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases , 2005 .

[12]  David Adam,et al.  Citation analysis: The counting house , 2002, Nature.

[13]  Katherine W. McCain,et al.  Visualizing a discipline: an author co-citation analysis of information science, 1972–1995 , 1998 .

[14]  Charles Oppenheim,et al.  Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE , 2003, J. Documentation.

[15]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[16]  Blaise Cronin,et al.  Research Brief rates of Return to citation , 1996, J. Documentation.

[17]  Mark Sanderson,et al.  Revisiting h measured on UK LIS and IR academics , 2008, J. Assoc. Inf. Sci. Technol..

[18]  C. Borgman,et al.  Scholarly Communication and Bibliometrics. , 1992 .

[19]  Andrew F. Monk,et al.  Positioning HCI: Journals, descriptors and parent disciplines , 1998, Behav. Inf. Technol..

[20]  Rafael Ball,et al.  Science indicators revisited - Science Citation Index versus SCOPUS: A bibliometric comparison of both citation databases , 2007, Inf. Serv. Use.

[21]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[22]  Henk F. Moed,et al.  Developing Bibliometric Indicators of Research Performance in Computer Science , 2007 .

[23]  Judit Bar-Ilan,et al.  An ego-centric citation analysis of the works of Michael O. Rabin based on multiple citation indexes , 2006, Inf. Process. Manag..

[24]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[25]  Jean Tirole,et al.  Evaluating Economics Research in Europe: An Introduction , 2003 .

[26]  Charles Oppenheim,et al.  Using the h -index to rank influential British researchers in information science and librarianship: Brief Communication , 2007 .

[27]  Julian Warner,et al.  A critical review of the application of citation studies to the Research Assessment Exercises , 2000, J. Inf. Sci..

[28]  Mike Thelwall,et al.  Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines , 2008, Scientometrics.

[29]  Lokman I. Meho,et al.  Using the h-index to rank influential information scientistss: Brief Communication , 2006 .

[30]  R. Sacks.  Phone search : The search goes on , 2006 .

[31]  C. Lee Giles,et al.  Scholarly publishing in the Internet age: a citation analysis of computer science literature , 2001, Inf. Process. Manag..

[32]  K. McCain,et al.  Visualization of Literatures. , 1997 .

[33]  Henry G. Small,et al.  Visualizing Science by Citation Mapping , 1999, J. Am. Soc. Inf. Sci..

[34]  Eugene Garfield,et al.  Citation indexing - its theory and application in science, technology, and humanities , 1979 .

[35]  Henk F. Moed,et al.  The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review , 2007 .

[36]  Jonathan Furner,et al.  Scholarly communication and bibliometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[37]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[38]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[39]  Charles Oppenheim,et al.  Comparing alternatives to the Web of Science for coverage of the social sciences' literature , 2007, J. Informetrics.

[40]  Ben Shneiderman,et al.  Building HCI partnerships and infrastructure , 1993, Behav. Inf. Technol..

[41]  Dana L. Roth,et al.  The emergence of competitors to the Science Citation Index and the Web of Science , 2005 .

[42]  Gabriel Pinski,et al.  Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics , 1976, Inf. Process. Manag..

[43]  Péter Jacsó,et al.  Deflated, inflated and phantom citation counts , 2006, Online Inf. Rev..

[44]  Thomas W. Dillon,et al.  Mapping the discourse of HCI researchers with citation analysis , 1995, SGCH.

[45]  Judit Bar-Ilan,et al.  Informetrics at the beginning of the 21st century - A review , 2008, J. Informetrics.

[46]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[47]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[48]  Alireza Noruzi.  Google Scholar: The New Generation of Citation Indexes , 2005 .

[49]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[50]  James Testa,et al.  The Thomson Scientific journal selection process. , 2006, International microbiology : the official journal of the Spanish Society for Microbiology.

[51]  Robert D. Galliers,et al.  An Anatomy of European Information Systems Research ECIS 1993-ECIS 2002 : Some Initial Findings , 2002, ECIS.

[52]  Judit Bar-Ilan,et al.  Some measures for comparing citation databases , 2007, J. Informetrics.

[53]  Lokman I. Meho,et al.  The shifting balance of intellectual trade in information studies , 2008, J. Assoc. Inf. Sci. Technol..

[54]  Christoph Neuhaus,et al.  Google Scholar Goes to School: The Presence of Google Scholar on College and University Web Sites , 2008 .

[55]  Charles Oppenheim,et al.  Using the h-index to rank influential British researchers in information science and librarianship , 2007, J. Assoc. Inf. Sci. Technol..

[56]  Nisa Bakkalbasi,et al.  An Examination of Citation Counts in a New Scholarly Communication Environment , 2005, D Lib Mag..

[57]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[58]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises , 1996, Scientometrics.

[59]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[60]  Tom Carey,et al.  ACM SIGCHI Curricula for Human-Computer Interaction , 1992 .

[61]  Katherine W. McCain,et al.  Visualizing a Discipline: An Author Co-Citation Analysis of Information Science, 1972-1995 , 1998, J. Am. Soc. Inf. Sci..

[62]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..