Using Google Scholar in research evaluation of humanities and social science programs: A comparison with Web of Science data

In this paper, we report on the application of Google Scholar (GS)-based metrics in the formal assessment of research programs. Involved were programs in the fields of Education, Pedagogical Sciences, and Anthropology in The Netherlands. Also, a comparative analysis has been conducted of the results based on GS and Web of Science (WoS). Studies critical of GS point at its reliability of data. We show how the reliability of the GS data for the bibliometric analysis of the assessment can be improved by excluding non-verifiable citing sources from the full second-order GS citing data. The study of the background of these second-order sources demonstrates a broadening of the citing sources. The comparison of GS with WoS citations for the publications of the programs shows that it is promising to use GS for fields with lower degrees of coverage in WoS, in particular for fields that produce more diverse types of output than just research articles. Restrictions to the use of GS are the intensive manual data handling and cleaning, necessary for a feasible and proper data collection. We discuss wider implications of the findings for bibliometric analysis and for the practices and policies in research evaluation.

[1]  V. Larivière.  The limits of bibliometrics for the analysis of the social sciences and humanities literature , 2010 .

[2]  Ludo Waltman,et al.  Counting publications and citations: Is more always better? , 2013, J. Informetrics.

[3]  Vincent Larivière,et al.  Improving the coverage of social science and humanities researchers' output: The case of the Érudit journal platform , 2011, J. Assoc. Inf. Sci. Technol..

[4]  A. Kulkarni,et al.  Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. , 2009, JAMA.

[5]  Judit Bar-Ilan,et al.  Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar , 2010, Scientometrics.

[6]  Thed N. van Leeuwen,et al.  Highly cited non-journal publications in political science, economics and psychology: a first exploration , 2010, Scientometrics.

[7]  John Mingers,et al.  Counting the citations: a comparison of Web of Science and Google Scholar in the field of business and management , 2010, Scientometrics.

[8]  Rafael Ruiz-Pérez,et al.  Google Scholar como herramienta para la evaluación científica. , 2009 .

[9]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[10]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[11]  Giovanni Abramo,et al.  Inefficiency in selecting products for submission to national research assessment exercises , 2013, Scientometrics.

[12]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[13]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[14]  T. V. Leeuwen.  Bibliometric research evaluations, Web of Science and the Social Sciences and Humanities: a problematic relationship? , 2013 .

[15]  Joost C. F. de Winter,et al.  The expansion of Google Scholar versus Web of Science: a longitudinal study , 2013, Scientometrics.

[16]  Péter Jacsó,et al.  Metadata mega mess in Google Scholar , 2010, Online Inf. Rev..

[17]  D. Hicks.  The Four Literatures of Social Science , 2004 .

[18]  Mike Thelwall,et al.  Can the impact of non‐Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia , 2014, J. Assoc. Inf. Sci. Technol..

[19]  Mike Thelwall,et al.  Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines , 2008, Scientometrics.

[20]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[21]  Péter Jacsó,et al.  Deflated, inflated and phantom citation counts , 2006, Online Inf. Rev..

[22]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[23]  Isidro F. Aguillo.  Is Google Scholar useful for bibliometrics? A webometric analysis , 2012, Scientometrics.

[24]  Richard Van Noorden.  Google Scholar pioneer on search engine’s future , 2014 .

[25]  Nicolás Robinson-García,et al.  The Google scholar experiment: How to index false papers and manipulate bibliometric indicators , 2013, J. Assoc. Inf. Sci. Technol..

[26]  Thed N. van Leeuwen,et al.  Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university , 2015, Scientometrics.

[27]  Massimo Franceschet,et al.  A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar , 2010, Scientometrics.

[28]  Anne-Wil Harzing,et al.  Google Scholar: the democratization of citation analysis , 2007 .

[29]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[30]  Anne-Wil Harzing,et al.  A longitudinal study of Google Scholar coverage between 2012 and 2013 , 2013, Scientometrics.

[31]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007 .

[32]  Lei Wang,et al.  Three options for citation tracking: Google Scholar, Scopus and Web of Science , 2006, Biomedical digital libraries.

[33]  Nicolás Robinson-García,et al.  Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting , 2012, ArXiv.