Inter-rater reliability of h-index scores calculated by Web of Science and Scopus for clinical epidemiology scientists.

OBJECTIVE We investigated the inter-rater reliability of Web of Science (WoS) and Scopus when calculating the h-index of 25 senior scientists in the Clinical Epidemiology Program of the Ottawa Hospital Research Institute. MATERIALS AND METHODS Bibliometric information and the h-indices for the subjects were computed by four raters using the automatic calculators in WoS and Scopus. Correlation and agreement between ratings was assessed using Spearman's correlation coefficient and a Bland-Altman plot, respectively. RESULTS Data could not be gathered from Google Scholar due to feasibility constraints. The Spearman's rank correlation between the h-index of scientists calculated with WoS was 0.81 (95% CI 0.72-0.92) and with Scopus was 0.95 (95% CI 0.92-0.99). The Bland-Altman plot showed no significant rater bias in WoS and Scopus; however, the agreement between ratings is higher in Scopus compared to WoS. CONCLUSION Our results showed a stronger relationship and increased agreement between raters when calculating the h-index of a scientist using Scopus compared to WoS. The higher inter-rater reliability and simple user interface used in Scopus may render it the more effective database when calculating the h-index of senior scientists in epidemiology.

[1]  Alejandro M. Aragón,et al.  A measure for the impact of research , 2013, Scientific Reports.

[2]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[3]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[4]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[5]  Teodor P. Grantcharov,et al.  The h-index outperforms other bibliometrics in the assessment of research performance in general surgery: a province-wide study. , 2013, Surgery.

[6]  Martin Ravallion,et al.  On measuring scholarly influence by citations , 2011, Scientometrics.

[7]  R. Janke,et al.  The academic librarian as co-investigator on an interprofessional primary research team: a case study. , 2014, Health information and libraries journal.

[8]  Jingfeng Xia,et al.  Positioning Open Access Journals in a LIS Journal Ranking , 2012, Coll. Res. Libr..

[9]  L. Bornmann,et al.  The state of h index research , 2009, EMBO reports.

[10]  T. Grantcharov,et al.  h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance. , 2011, British journal of anaesthesia.

[11]  Lutz Bornmann,et al.  What do we know about the h index? , 2007, J. Assoc. Inf. Sci. Technol..

[12]  D. Hodge,et al.  RANKING DISCIPLINARY JOURNALS WITH THE GOOGLE SCHOLAR H-INDEX: A NEW TOOL FOR CONSTRUCTING CASES FOR TENURE, PROMOTION, AND OTHER PROFESSIONAL DECISIONS , 2011 .

[13]  Viren N. Naik,et al.  Within-Team Debriefing Versus Instructor-Led Debriefing for Simulation-Based Education: A Randomized Controlled Trial , 2013, Annals of surgery.

[14]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[15]  Péter Jacsó,et al.  The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases , 2008, Online Inf. Rev..

[16]  Yvonne Rogers,et al.  Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science , 2008, J. Assoc. Inf. Sci. Technol..

[17]  Salih Selek,et al.  Use of h index and g index for American academic psychiatry , 2013, Scientometrics.

[18]  Ara Darzi,et al.  Measuring Academic Performance for Healthcare Researchers with the H Index: Which Search Tool Should Be Used? , 2012, Medical Principles and Practice.

[19]  K. Denninghoff,et al.  The distribution of the h-index among academic emergency physicians in the United States. , 2013, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[20]  Jan Hauke,et al.  Comparison of Values of Pearson's and Spearman's Correlation Coefficients on the Same Sets of Data , 2011 .

[21]  S. D. De Groote,et al.  Coverage of Google Scholar, Scopus, and Web of Science: a case study of the h-index in nursing. , 2012, Nursing outlook.