The h-index outperforms other bibliometrics in the assessment of research performance in general surgery: a province-wide study.

BACKGROUND The h-index is used as an objective measure of research impact. Its validity, however, is not known in the context of general surgery and comparisons with other bibliometric indices are lacking. We sought to evaluate the h-index as a reliable and valid measure of research performance in general surgery across 6 universities in the province of Ontario, Canada. METHODS Bibliometric indices for 219 faculty members in general surgery were calculated using the Scopus and Web of Science online databases. We investigated agreement between the databases. A 2-way analysis of variance was used to compare the h-index of surgeons grouped by institutional affiliation and academic rank and to identify the relative impact of these factors on different bibliometric indices. RESULTS The agreement on h-indices between the Scopus and Web of Science was problematic. The h-index was associated more strongly with academic rank (academic rank accounted for 33.3% of researcher's h-index) than of the number of publications (12.5%) or the number of citations per author (10.2%). The number of citations per paper was not associated with academic rank. The institutional affiliation affected bibliometric indices to a similar degree to academic rank. CONCLUSION Our data suggest better construct validity for the h-index than for other bibliometrics, although the agreement of h-index values between databases can be problematic for some researchers. The use of the h-index as a criterion-based assessment across different universities is problematic and that it should be used as a normative assessment tool, with comparisons with a specified population of interest.

[1]  D. Hoaglin,et al.  Chairpersons of pathology in the United States: limited benchmarks for publications. , 2010, American journal of clinical pathology.

[2]  Charles R. Thomas,et al.  Estimation of citation-based scholarly activity among radiation oncology faculty at domestic residency-training institutions: 1996-2007. , 2009, International journal of radiation oncology, biology, physics.

[3]  Philip Ball,et al.  Achievement index climbs the ranks , 2007, Nature.

[4]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[5]  Joseph Foss,et al.  Comparing Methods of Clinical Measurement: Reporting Standards for Bland and Altman Analysis , 2000, Anesthesia and analgesia.

[6]  Régis Beuscart,et al.  From bibliometric analysis to research policy: the use of SIGAPS in Lille University Hospital , 2006, MIE.

[7]  A. Lozano,et al.  Academic impact and rankings of American and Canadian neurosurgical departments as assessed using the h index. , 2010, Journal of neurosurgery.

[8]  A. Kulkarni,et al.  Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. , 2009, JAMA.

[9]  M. Kane.  Current Concerns in Validity Theory , 2001 .

[10]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[11]  S. Bhayani,et al.  Does citation analysis reveal association between h-index and academic rank in urology? , 2009, Urology.

[12]  Charles R. Thomas,et al.  Bibliometric analysis of radiation oncology departmental scholarly publication productivity at domestic residency training institutions. , 2009, Journal of the American College of Radiology : JACR.

[13]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[14]  R. Wears,et al.  Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. , 2002, JAMA.

[15]  W. Couldwell,et al.  Use of the h index in neurosurgery. Clinical article. , 2009, Journal of neurosurgery.

[16]  T. Grantcharov,et al.  h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance. , 2011, British journal of anaesthesia.

[17]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.