Accuracy and completeness of publication and citation records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of h indices in Psychology

Hirsch’s h index is becoming the standard measure of an individual’s research accomplishments. The aggregation of individuals’ measures is also the basis for global measures at institutional or national levels. To investigate whether the h index can be reliably computed through alternative sources of citation records, the Web of Science (WoS), PsycINFO and Google Scholar (GS) were used to collect citation records for known publications of four Spanish psychologists. Compared withWoS, PsycINFO included a larger percentage of publication records, whereas GS outperformed WoS and PsycINFO in this respect. Compared with WoS, PsycINFO retrieved a larger number of citations in unique areas of psychology, but it retrieved a smaller number of citations in areas that are close to statistics or the neurosciences, whereas GS retrieved the largest numbers of citations in all cases. Incorrect citations were scarce inWos (0.3%),more prevalent in PsycINFO (1.1%), and overwhelming in GS (16.5%). All platforms retrieved unique citations, the largest set coming from GS. WoS and PsycINFO cover distinct areas of psychology unevenly, thus applying different penalties on the h index of researches working in different fields. Obtaining fair and accurate h indices required the union of citations retrieved by all three platforms.

[1]  Miguel A Hernán,et al.  Epidemiologists (of all people) should question journal impact factors. , 2008, Epidemiology.

[2]  Miquel Porta,et al.  How come scientists uncritically adopt and embody Thomson's bibliographic impact factor? , 2008, Epidemiology.

[3]  Blaise Cronin,et al.  Comparative citation rankings of authors in monographic and journal literature: a study of sociology , 1997, J. Documentation.

[4]  P. Seglen,et al.  Citations and journal impact factors: questionable indicators of research quality , 1997, Allergy.

[5]  Roger A Brumback.  Impact Factor Wars: Episode V—The Empire Strikes Back , 2009, Journal of child neurology.

[6]  M. Falagas,et al.  Editors may inappropriately influence authors' decisions regarding selection of references in scientific articles , 2007, International Journal of Impotence Research.

[7]  R. Brumback,et al.  Worshiping False Idols: The Impact Factor Dilemma , 2008, Journal of child neurology.

[8]  Charles Oppenheim,et al.  Comparing alternatives to the Web of Science for coverage of the social sciences' literature , 2007, J. Informetrics.

[9]  The Impact Factor Game , 2006, PLoS medicine.

[10]  Guang Yu,et al.  The self-cited rate of scientific journals and the manipulation of their impact factors , 2007, Scientometrics.

[11]  A. Kulkarni,et al.  Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. , 2009, JAMA.

[12]  A. Baneyx,et al.  “Publish or Perish” as citation metrics used to analyze scientific output in the humanities: International case studies in economics, geography, social sciences, philosophy, and history , 2008, Archivum Immunologiae et Therapiae Experimentalis.

[13]  Péter Jacsó,et al.  Deflated, inflated and phantom citation counts , 2006, Online Inf. Rev..

[14]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Web of Science , 2008, Online Inf. Rev..

[15]  P. V. van Diest,et al.  Impactitis: new cures for an old disease. , 2001, Journal of clinical pathology.

[16]  Ronald Rousseau,et al.  The influence of missing publications on the Hirsch index , 2007, J. Informetrics.

[17]  Nicole Haeffner-Cavaillon,et al.  The use of bibliometric indicators to help peer-review assessment , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[18]  A. Wilcox,et al.  Rise and fall of the Thomson impact factor. , 2008, Epidemiology.

[19]  E. Garfield.  Journal impact factor: a brief review. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[20]  Michael Levine-Clark,et al.  A comparative analysis of social sciences citation tools , 2009, Online Inf. Rev..

[21]  P. Lawrence.  The mismeasurement of science , 2007, Current Biology.

[22]  Miguel A García-Pérez,et al.  The Hirsch h Index in a Non-Mainstream Area: Methodology of the Behavioral Sciences in Spain , 2009, The Spanish journal of psychology.

[23]  Hans-Dieter Daniel,et al.  Data sources for performing citation analysis: an overview , 2008, J. Documentation.

[24]  L. Bornmann,et al.  The state of h index research , 2009, EMBO reports.

[25]  Yu-Wei Chang,et al.  Characteristics of research output in social sciences and humanities: From a research evaluation perspective , 2008, J. Assoc. Inf. Sci. Technol..

[26]  Marcia Henry,et al.  Citation searching : New players, new tools , 2006 .

[27]  Chris Neuhaus,et al.  The Depth and Breadth of Google Scholar: An Empirical Study , 2006 .

[28]  Emma Hill,et al.  Show me the data. , 1998 .

[29]  Tove Faber Frandsen,et al.  Intradisciplinary differences in database coverage and the consequences for bibliometric research , 2008, J. Assoc. Inf. Sci. Technol..

[30]  Elizabeth S. Vieira,et al.  A comparison of Scopus and Web of Science for a typical university , 2009, Scientometrics.

[31]  Jesús F. Salgado,et al.  La productividad científica y el índice h de Hirchs de la psicología social española: convergencia entre indicadores de productividad y comparación con otras áreas , 2007 .

[32]  Lei Wang,et al.  Three options for citation tracking: Google Scholar, Scopus and Web of Science , 2006, Biomedical digital libraries.

[33]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[34]  Michael Grüninger,et al.  Introduction , 2002, CACM.

[35]  S. Sala,et al.  Multi-authors' self-citation: A further impact factor bias? , 2008, Cortex.

[36]  Ignasi Carrió.  Of impact, metrics and ethics , 2008, European Journal of Nuclear Medicine and Molecular Imaging.

[37]  Stevan Harnad,et al.  Validating research performance metrics against peer rankings , 2008 .

[38]  Judit Bar-Ilan,et al.  An ego-centric citation analysis of the works of Michael O. Rabin based on multiple citation indexes , 2006, Inf. Process. Manag..

[39]  Philip Campbell,et al.  Escape from the impact factor , 2008 .

[40]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[41]  Lars Iselid,et al.  Web of Science and Scopus: a journal title overlap study , 2008, Online Inf. Rev..

[42]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[43]  Nisa Bakkalbasi,et al.  An Examination of Citation Counts in a New Scholarly Communication Environment , 2005, D Lib Mag..

[44]  A. A. Waheed.  Citation rate unrelated to journals' impact factors , 2003, Nature.

[45]  John Mingers,et al.  Measuring the research contribution of management academics using the Hirsch-index , 2009, J. Oper. Res. Soc..

[46]  Péter Jacsó,et al.  The h-index for countries in Web of Science and Scopus , 2009, Online Inf. Rev..

[47]  Mike Thelwall,et al.  Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines , 2008, Scientometrics.

[48]  Miguel A. García-Pérez,et al.  The Decade 1989–1998 in Spanish Psychology: An Analysis of Research in Statistics, Methodology, and Psychometric Theory , 2001, The Spanish Journal of Psychology.

[49]  Les Carr,et al.  The Open Research Web: A Preview of the Optimal and the Inevitable , 2006 .

[50]  Janne S. Kotiaho,et al.  Papers vanish in mis-citation black hole , 1999, Nature.

[51]  Peter Jasco,et al.  Testing the Calculation of a Realistic h-index in Google Scholar, Scopus, and Web of Science for F. W. Lancaster , 2008 .

[52]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Scopus , 2008, Online Inf. Rev..

[53]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Google Scholar , 2008, Online Inf. Rev..

[54]  Janne S. Kotiaho,et al.  Unfamiliar citations breed mistakes , 1999, Nature.

[55]  Mark Sanderson,et al.  Revisiting h measured on UK LIS and IR academics , 2008, J. Assoc. Inf. Sci. Technol..

[56]  Kai Simons,et al.  The Misused Impact Factor , 2008, Science.

[57]  Péter Jacsó,et al.  Comparison and Analysis of the Citedness Scores in Web of Science and Google Scholar , 2005, ICADL.

[58]  Miguel A. García-Pérez,et al.  A multidimensional extension to Hirsch’s h-index , 2009, Scientometrics.

[59]  Andrzej K. Wróblewski,et al.  A commentary on misuses of the impact factor , 2008, Archivum Immunologiae et Therapiae Experimentalis.

[60]  Brian D. Cameron,et al.  Trends in the Usage of ISI Bibliometric Data: Uses, Abuses, and Implications , 2005 .

[61]  Lee F Rogers,et al.  Impact factor: the numbers game. , 2002, AJR. American journal of roentgenology.

[62]  Nigel L. Brown.  On the trail of the prolific Dr Path , 1999, Nature.

[63]  Henk F. Moed,et al.  Is the impact of journal impact factors decreasing? , 2008, J. Documentation.

[64]  Denise Beaubien Bennett,et al.  Name Authority Challenges for Indexing and Abstracting Databases , 2006 .

[65]  William H. Walters,et al.  Google Scholar Search Performance: Comparative Recall and Precision , 2009 .

[66]  A. Sevinç,et al.  Manipulating impact factor: an unethical issue or an Editor's choice? , 2004, Swiss medical weekly.

[67]  Mary Shultz,et al.  Comparing test searches in PubMed and Google Scholar. , 2007, Journal of the Medical Library Association : JMLA.

[68]  Peretz Lavie,et al.  The race for the impact factor , 2009, Journal of sleep research.

[69]  G Williams,et al.  Misleading, unscientific, and unjust: the United Kingdom's research assessment exercise , 1998, BMJ.

[70]  Sune Lehmann,et al.  A quantitative analysis of indicators of scientific performance , 2008, Scientometrics.

[71]  R. Ladle,et al.  Hidden dangers of a 'citation culture' , 2008 .

[72]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[73]  Henk F. Moed,et al.  New developments in the use of citation analysis in research evaluation , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[74]  R. Brumback,et al.  Response to Correspondence, “`Worshiping False Idols: The Impact Factor Dilemma': Correcting the Record” , 2008 .

[75]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[76]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[77]  N. C. Price,et al.  What's in a name (or a number or a date)? , 1998, Nature.

[78]  P. Jacsó.  As we may search : Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases , 2005 .

[79]  Richard Smith.  Unscientific practice flourishes in science , 1998, BMJ.

[80]  Michael Levine-Clark,et al.  A Comparative Citation Analysis of Web of Science, Scopus, and Google Scholar , 2008 .

[81]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[82]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[83]  K S Joseph,et al.  CMAJ's impact factor: room for recalculation. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[84]  H. K. Schutte,et al.  Reaction of Folia Phoniatrica et Logopaedica on the Current Trend of Impact Factor Measures , 2007, Folia Phoniatrica et Logopaedica.

[85]  Matthew E. Falagas,et al.  The top-ten in journal impact factor manipulation , 2008, Archivum Immunologiae et Therapiae Experimentalis.

[86]  Katherine M. Whitley,et al.  Analysis of Scifinder Scholar and Web of Science citation searches , 2002, J. Assoc. Inf. Sci. Technol..

[87]  Péter Jacsó,et al.  Software issues related to cited references , 2007, Online Inf. Rev..

[88]  Daniel Pauly,et al.  Equivalence of results from two citation analyses: Thomson ISI's Citation Index and Google's Scholar service , 2005 .

[89]  Péter Jacsó,et al.  Errors of omission and their implications for computing scientometric measures in evaluating the publishing productivity and impact of countries , 2009, Online Inf. Rev..

[90]  Robert Schroeder,et al.  Pointing Users Toward Citation Searching: Using Google Scholar and Web of Science , 2007 .

[91]  Yvonne Rogers,et al.  Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science , 2008, J. Assoc. Inf. Sci. Technol..

[92]  A. Agrawal,et al.  Corruption of journal Impact Factors. , 2005, Trends in ecology & evolution.

[93]  Emma Hill,et al.  Irreproducible Results—a Response to Thomson Scientific , 2008, The Journal of general physiology.

[94]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[95]  Dana L. Roth,et al.  The emergence of competitors to the Science Citation Index and the Web of Science , 2005 .

[96]  William H. Walters,et al.  Google Scholar coverage of a multidisciplinary field , 2007, Inf. Process. Manag..

[97]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[98]  D. Colquhoun.  Challenging the tyranny of impact factors , 2003, Nature.

[99]  Péter Jacsó,et al.  Google Scholar revisited , 2008, Online Inf. Rev..

[100]  María Peñaranda Ortega,et al.  Consecuencias de los errores en las referencias bibliográficas. El caso de la revista Psicothema , 2009 .

[101]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[102]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..