Can we use Google Scholar to identify highly-cited documents?

The main objective of this paper is to empirically test whether the identification of highly-cited documents through Google Scholar is feasible and reliable. To this end, we carried out a longitudinal analysis (1950–2013), running a generic query (filtered only by year of publication) to minimise the effects of academic search engine optimisation. This gave us a final sample of 64,000 documents (1000 per year). The strong correlation between a document’s citations and its position in the search results (r=−0.67) led us to conclude that Google Scholar is able to identify highly-cited papers effectively. This, combined with Google Scholar’s unique coverage (no restrictions on document type and source), makes the academic search engine an invaluable tool for bibliometric research relating to the identification of the most influential scientific documents. We find evidence, however, that Google Scholar ranks those documents whose language (or geographical web domain) matches with the user’s interface language higher than could be expected based on citations. Nonetheless, this language effect and other factors related to the Google Scholar’s operation, i.e. the proper identification of versions and the date of publication, only have an incidental impact. They do not compromise the ability of Google Scholar to identify the highly-cited papers.

[1]  Judit Bar-Ilan,et al.  Some measures for comparing citation databases , 2007, J. Informetrics.

[2]  Joseph R. Kraus,et al.  Finding Chemistry Information Using Google Scholar , 2007 .

[3]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[4]  Gentiane Haesbroeck,et al.  The Utility of Google Scholar When Searching Geographical Literature: Comparison With Three Commercial Bibliographic Databases , 2015 .

[5]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[6]  Julie Carpenter,et al.  Researchers of Tomorrow: The research behaviour of Generation Y doctoral students , 2012, Inf. Serv. Use.

[7]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Scopus , 2008, Online Inf. Rev..

[8]  Lokman I. Meho,et al.  Citation Analysis: A Comparison of Google Scholar, Scopus, and Web of Science , 2007, Proceedings of the American Society for Information Science and Technology.

[9]  Emilio Delgado López-Cózar,et al.  Google Scholar e índice h en biomedicina: la popularización de la evaluación bibliométrica , 2013 .

[10]  Declan Butler,et al.  Science searches shift up a gear as Google starts Scholar engine , 2004, Nature.

[11]  David Ettinger.  The Triumph of Expediency: The Impact of Google Scholar on Library Instruction , 2008 .

[12]  Péter Jacsó,et al.  Testing the Calculation of a Realistic h-index in Google Scholar, Scopus, and Web of Science for F. W. Lancaster , 2008, Libr. Trends.

[13]  Lei Wang,et al.  Three options for citation tracking: Google Scholar, Scopus and Web of Science , 2006, Biomedical digital libraries.

[14]  C. Nyquist,et al.  An Academic Librarian's Response to the “ITHAKA Faculty Survey 2009: Key Strategic Insights for Libraries, Publishers, and Societies” , 2010 .

[15]  Joost C. F. de Winter,et al.  The expansion of Google Scholar versus Web of Science: a longitudinal study , 2013, Scientometrics.

[16]  Fiorenzo Franceschini,et al.  Empirical analysis and classification of database errors in Scopus and Web of Science , 2016, J. Informetrics.

[17]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007 .

[18]  Jöran Beel,et al.  Google Scholar’s Ranking Algorithm : An Introductory Overview , 2009 .

[19]  Jöran Bela Erik Beel,et al.  Academic Search Engine Optimization (ASEO ): Optimizing Scholarly Literature for Google Scholar & Co. , 2010 .

[20]  Isidro F. Aguillo.  Is Google Scholar useful for bibliometrics? A webometric analysis , 2012, Scientometrics.

[21]  Anne-Wil Harzing,et al.  Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison , 2015, Scientometrics.

[22]  Jared L. Howland,et al.  How Scholarly Is Google Scholar? A Comparison to Library Databases , 2009 .

[23]  Richard Van Noorden.  Online collaboration: Scientists and the social network , 2014, Nature.

[24]  Bela Gipp,et al.  Academic Search Engine Spam and Google Scholar's Resilience Against it , 2010 .

[25]  Roger C. Schonfeld,et al.  UK survey of academics 2015 , 2013 .

[26]  Thomas W. Conkling,et al.  Google Scholar’s Coverage of the Engineering Literature: An Empirical Study , 2008 .

[27]  Nicolás Robinson-García,et al.  The Google scholar experiment: How to index false papers and manipulate bibliometric indicators , 2013, J. Assoc. Inf. Sci. Technol..

[28]  Mike Thelwall,et al.  Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines , 2008, Scientometrics.

[29]  Dirk Lewandowski,et al.  Problems with the use of web search engines to find results in foreign languages , 2008, Online Inf. Rev..

[30]  Susan Gardner,et al.  Gaga over Google? Scholar in the Social Sciences , 2005 .

[31]  Judit Bar-Ilan,et al.  Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar , 2010, Scientometrics.

[32]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Google Scholar , 2008, Online Inf. Rev..

[33]  Jerri L. Ledford,et al.  SEO: Search Engine Optimization Bible , 2007 .

[34]  Madian Khabsa,et al.  Digital commons , 2020, Internet Policy Rev..

[35]  Jim Giles,et al.  Science in the web age: Start your engines , 2005, Nature.

[36]  Stéfan Jacques Darmoni,et al.  Is the coverage of google scholar enough to be used alone for systematic reviews , 2013, BMC Medical Informatics and Decision Making.

[37]  Dean Giustini,et al.  How Google is changing medicine , 2005, BMJ : British Medical Journal.

[38]  Enrique Orduña-Malea,et al.  Back to the past: on the shoulders of an academic search engine giant , 2016, Scientometrics.

[39]  Anne-Wil Harzing,et al.  A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners , 2013, Scientometrics.

[40]  Isidro F. Aguillo,et al.  Cibermetría : midiendo el espacio red , 2014 .

[41]  Enrique Orduña-Malea,et al.  Methods for estimating the size of Google Scholar , 2014, Scientometrics.

[42]  Michael P. Evans,et al.  Analysing Google rankings through search engine optimization data , 2007, Internet Res..

[43]  Péter Jacsó,et al.  Deflated, inflated and phantom citation counts , 2006, Online Inf. Rev..

[44]  Rafael Ruiz-Pérez,et al.  Google Scholar como herramienta para la evaluación científica. , 2009 .

[45]  José Luis Ortega.  Academic search engines : a quantitative outlook , 2014 .

[46]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[47]  Dean Giustini,et al.  The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews , 2013, Systematic Reviews.

[48]  Anne-Wil Harzing,et al.  A longitudinal study of Google Scholar coverage between 2012 and 2013 , 2013, Scientometrics.

[49]  Péter Jacsó,et al.  Academic Search Engines: A Quantitative Outlook , 2015, Online Inf. Rev..

[50]  Mike Thelwall,et al.  Search markets and search results: The case of Bing , 2013 .

[51]  Péter Jacsó,et al.  Using Google Scholar for journal impact factors and the h‐index in nationwide publishing assessments in academia – siren songs and air‐raid sirens , 2012 .

[52]  Eugene Barsky,et al.  A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations , 2005 .

[53]  Enrique Orduña-Malea,et al.  Proceedings Scholar Metrics: H Index of proceedings on Computer Science, Electrical & Electronic Engineering, and Communications according to Google Scholar Metrics (2009-2013) , 2014, ArXiv.

[54]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[55]  Enrique Orduña-Malea,et al.  Google Scholar Metrics evolution: an analysis according to languages , 2013, Scientometrics.

[56]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..