Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors

With the introduction of an increasing number of evaluation indexes, researchers have begun to pay attention to the limitations of such indexes in research evaluation, understanding which to avoid misusing and making evaluation more scientific and reasonable. Analysing the principles of the h-index, g-index, AR-index, p-index, integrated impact indicator (I3), and academic trace, this paper explores their limitations in measuring the research performance of authors from the perspectives of consistency, the degree of discrimination, and the statistical relationship between the values of indicators and the number of publications and citations. There are some interesting findings. These six indicators are highly consistent, and they are all more susceptible to the number of publications than to the frequency of citations. Among them, the h-index has the lowest degree of discrimination, followed by the g-index, I3, AR-index, p-index, and academic trace. The g-index ignores papers and citations other than the g-core. Moreover, compared to the h-index, the accumulation of citations makes it easier for the g-index to be equal to the number of papers published by an author, and once its value equals the number of papers, subsequent citations received by these papers will no longer contribute to the growth of the g-index unless the author publishes a new paper. Additionally, the AR-index ignores the h-tail papers and citations, which underestimates the impact of many researchers. Moreover, the p-index is insensitive to highly cited papers. Furthermore, the I3 is very vulnerable to the influence of the extremums in a data set. Finally, we propose considerations and suggestions for the research performance evaluation of authors.

[1]  Mônica G. Campiteli,et al.  An index to quantify an individual's scientific research valid across disciplines , 2005 .

[2]  Loet Leydesdorff,et al.  An Integrated Impact Indicator (I3): A New Definition of "Impact" with Policy Relevance , 2012, ArXiv.

[3]  Lutz Bornmann,et al.  A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants , 2011, J. Informetrics.

[4]  Juan E. Iglesias,et al.  Scaling the h-index for different scientific ISI fields , 2006, Scientometrics.

[5]  Loet Leydesdorff,et al.  Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative research design with policy implications , 2011, J. Assoc. Inf. Sci. Technol..

[6]  Lutz Bornmann,et al.  h-Index research in scientometrics: A summary , 2014, J. Informetrics.

[7]  Chun-Ting Zhang,et al.  The h’-Index, Effectively Improving the h-Index Based on the Citation Distribution , 2013, PloS one.

[8]  Jorge E. Hirsch,et al.  An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship , 2009, Scientometrics.

[9]  Michael Schreiber,et al.  To share the fame in a fair way, hm modifies h for multi-authored manuscripts , 2008 .

[10]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[11]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[12]  A. Harlev,et al.  Bibliometrics: tracking research impact by selecting the appropriate metrics , 2016, Asian journal of andrology.

[13]  Loet Leydesdorff,et al.  h-based I3-type multivariate vectors: multidimensional indicators of publication and citation scores , 2017 .

[14]  Gangan Prathap,et al.  Is there a place for a mock h-index? , 2010, Scientometrics.

[15]  Mônica G. Campiteli,et al.  Is it possible to compare researchers with different scientific interests? , 2006, Scientometrics.

[16]  Dietmar Wolfram,et al.  A comparison of citer and citation-based measure outcomes for multiple disciplines , 2010, J. Assoc. Inf. Sci. Technol..

[17]  Liming Liang,et al.  h-index sequence and h-index matrix: Constructions and applications , 2006, Scientometrics.

[18]  Ronald Rousseau,et al.  A formal relation between the h-index of a set of articles and their I3 score , 2012, J. Informetrics.

[19]  D. Janzing,et al.  A single-shot measurement of the energy of product states in a translation invariant spin chain can replace any quantum computation , 2007, 0710.1615.

[20]  Vincent Larivière,et al.  There are neither "king" nor "crown" in scientometrics: Comments on a supposed "alternative" method of normalization , 2011, J. Informetrics.

[21]  Michael Schreiber,et al.  Revisiting the g-index: The average number of citations in the g-core , 2010, J. Assoc. Inf. Sci. Technol..

[22]  Philip Ball,et al.  Index aims for fair ranking of scientists , 2005, Nature.

[23]  Jorge E. Hirsch,et al.  hα: An index to quantify an individual’s scientific leadership , 2018, Scientometrics.

[24]  Chun-Ting Zhang,et al.  The e-Index, Complementing the h-Index for Excess Citations , 2009, PloS one.

[25]  Olle Persson.  'Citation Indexes for Science' - A 50 year citation history , 2005 .

[26]  Leo Egghe,et al.  The Hirsch index and related impact measures , 2010, Annu. Rev. Inf. Sci. Technol..

[27]  Richard S. J. Tol,et al.  A rational, successive g-index applied to economics departments in Ireland , 2008, J. Informetrics.

[28]  Dietmar Wolfram,et al.  Citer analysis as a measure of research impact: library and information science as a case study , 2010, Scientometrics.

[29]  Lutz Bornmann,et al.  How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers , 2013, J. Assoc. Inf. Sci. Technol..

[30]  Giovanni Abramo,et al.  The suitability of h and g indexes for measuring the research performance of institutions , 2013, Scientometrics.

[31]  Rodrigo Costas,et al.  Is g-index better than h-index? An exploratory study at the individual level , 2008, Scientometrics.

[32]  Giovanni Abramo,et al.  Assessing the accuracy of the h- and g-indexes for measuring researchers' productivity , 2013, J. Assoc. Inf. Sci. Technol..

[33]  Quentin L. Burrell,et al.  On the h-index, the size of the Hirsch core and Jin's A-index , 2007, J. Informetrics.

[34]  E. Garfield.  Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[35]  Gerhard J. Woeginger.  Generalizations of Egghe's g-index , 2009 .

[36]  Ronald Rousseau,et al.  Simple models and the corresponding h- and g-index , 2006 .

[37]  Michael Schreiber,et al.  An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the A-index, and the R-index , 2008, J. Assoc. Inf. Sci. Technol..

[38]  Francisco Herrera,et al.  hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices , 2010, Scientometrics.

[39]  L. Bornmann,et al.  A New Family of Cumulative Indexes for Measuring Scientific Performance , 2012, PloS one.

[40]  Yannis Manolopoulos,et al.  Generalized Hirsch h-index for disclosing latent facts in citation networks , 2007, Scientometrics.

[41]  Loet Leydesdorff,et al.  The “academic trace” of the performance matrix: A mathematical synthesis of the h‐index and the integrated impact indicator (I3) , 2013, J. Assoc. Inf. Sci. Technol..

[42]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[43]  Michael Schreiber,et al.  Twenty Hirsch index variants and other indicators giving more or less preference to highly cited papers , 2010, ArXiv.

[44]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[45]  Cameron Barnes.  The h-index Debate: An Introduction for Librarians , 2017 .

[46]  Michael Schreiber,et al.  Fractionalized counting of publications for the g-Index , 2009, J. Assoc. Inf. Sci. Technol..

[47]  Chun-Ting Zhang,et al.  A proposal for calculating weighted citations based on author rank , 2009, EMBO reports.