A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants

This paper presents the first meta-analysis of studies that computed correlations between the h index and variants of the h index (such as the g index; in total 37 different variants) that have been proposed and discussed in the literature. A high correlation between the h index and its variants would indicate that the h index variants hardly provide added information to the h index. This meta-analysis included 135 correlation coefficients from 32 studies. The studies were based on a total sample size of N=9005; on average, each study had a sample size of n=257. The results of a three-level cross-classified mixed-effects meta-analysis show a high correlation between the h index and its variants: Depending on the model, the mean correlation coefficient varies between .8 and .9. This means that there is redundancy between most of the h index variants and the h index. There is a statistically significant study-to-study variation of the correlation coefficients in the information they yield. The lowest correlation coefficients with the h index are found for the h index variants MII and m index. Hence, these h index variants make a non-redundant contribution to the h index.

[1]  Michael Schreiber.  A case study of the modified Hirsch index h m accounting for multiple coauthors , 2009 .

[2]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[3]  Michael Schreiber,et al.  A modification of the h-index: The hm-index accounts for multi-authored manuscripts , 2008, J. Informetrics.

[4]  Fiorenzo Franceschini,et al.  Analysis of the Hirsch index's operational properties , 2010, Eur. J. Oper. Res..

[5]  Sophia Rabe-Hesketh,et al.  Generalized latent variable models: multilevel, longitudinal, and structural equation models , 2004 .

[6]  A. Kinney.  National scientific facilities and their science impact on nonbiomedical research , 2007, Proceedings of the National Academy of Sciences.

[7]  Alex De Visscher,et al.  An index to measure a scientist's specific impact , 2010, J. Assoc. Inf. Sci. Technol..

[8]  Quentin L. Burrell,et al.  Hirsch index or Hirsch rate? Some thoughts arising from Liang’s data , 2007, Scientometrics.

[9]  G. A. Marcoulides.  Multilevel Analysis Techniques and Applications , 2002 .

[10]  John Antonakis,et al.  Quantifying Scholarly Impact: IQp Versus the Hirsch h , 2008 .

[11]  András Schubert,et al.  Hirsch-index for countries based on Essential Science Indicators data , 2007, Scientometrics.

[12]  Anthony F. J. van Raan.  Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups , 2013, Scientometrics.

[13]  John Antonakis,et al.  Quantifying Scholarly Impact: IQp Versus the Hirsch h , 2008, J. Assoc. Inf. Sci. Technol..

[14]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[15]  Peter Vinkler,et al.  The ź-index , 2009 .

[16]  Michael Schreiber,et al.  The influence of self‐citation corrections and the fractionalised counting of multi‐authored manuscripts on the Hirsch index , 2009 .

[17]  L. Bornmann,et al.  Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches , 2009 .

[18]  Adam R Hafdahl,et al.  Random-effects meta-analysis of correlations: Monte Carlo evaluation of mean estimators. , 2010, The British journal of mathematical and statistical psychology.

[19]  Lutz Bornmann,et al.  What do we know about the h index? , 2007, J. Assoc. Inf. Sci. Technol..

[20]  Ronald Rousseau,et al.  Properties of Hirsch-type indices: the case of library classification categories , 2009, Scientometrics.

[21]  J. Hirsch.  Does the h index have predictive power? , 2007, Proceedings of the National Academy of Sciences.

[22]  Bihui Jin.  The AR-index: complementing the h-index , 2007 .

[23]  Michael Schreiber.  A case study of the modified Hirsch index hm accounting for multiple coauthors , 2009, J. Assoc. Inf. Sci. Technol..

[24]  Richard S. J. Tol,et al.  Centres of research excellence in economics in the Republic of Ireland , 2007 .

[25]  H. Goldstein,et al.  The Graphical Presentation of a Collection of Means , 1995 .

[26]  Harvey Goldstein,et al.  Handbook of multilevel analysis , 2008 .

[27]  Alex De Visscher.  An index to measure a scientist's specific impact , 2010 .

[28]  Ronald Rousseau,et al.  Hirsch-type indices and library management: the case of Tongji University Library 1 , 2007 .

[29]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[30]  Yannis Manolopoulos,et al.  Generalized Hirsch h-index for disclosing latent facts in citation networks , 2007, Scientometrics.

[31]  John Panaretos,et al.  Assessing scientific research performance and impact with single indices , 2008, Scientometrics.

[32]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[33]  Mark Sanderson,et al.  Revisiting h measured on UK LIS and IR academics , 2008, J. Assoc. Inf. Sci. Technol..

[34]  Qiang Wu,et al.  The w-index: A measure to assess scientific impact by focusing on widely cited papers , 2010 .

[35]  G. Matt,et al.  What meta-analyses have and have not taught us about psychotherapy effects: a review and future directions. , 1997, Clinical psychology review.

[36]  Stan Lipovetsky,et al.  Generalized Latent Variable Modeling: Multilevel,Longitudinal, and Structural Equation Models , 2005, Technometrics.

[37]  Charles Oppenheim,et al.  The h-index: a broad review of a new bibliometric indicator , 2010, J. Documentation.

[38]  Lutz Bornmann,et al.  Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine , 2008, J. Assoc. Inf. Sci. Technol..

[39]  Yang Tao,et al.  A Study on Development Planning for Management Science and Engineering , 2006 .

[40]  William J. Browne,et al.  Non-Hierarchical Multilevel Models , 2008 .

[41]  Francisco Herrera,et al.  q2-Index: Quantitative and qualitative evaluation based on the number and impact of papers in the Hirsch core , 2010, J. Informetrics.

[42]  Michael Schreiber,et al.  Self-citation corrections for the Hirsch index , 2007 .

[43]  Lutz Bornmann,et al.  Are there really two types of h index variants? A validation study by using molecular life sciences data , 2009 .

[44]  Peter Vinkler,et al.  The π-index: a new indicator for assessing scientific impact , 2009, J. Inf. Sci..

[45]  Massimo Franceschet,et al.  A cluster analysis of scholar and journal bibliometric indicators , 2009, J. Assoc. Inf. Sci. Technol..

[46]  Loet Leydesdorff,et al.  Citation Environment of Angewandte Chemie , 2007, ArXiv.

[47]  R. Overton,et al.  A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator , 1998 .

[48]  M. Nahata,et al.  New indices in scholarship assessment. , 2009, American journal of pharmaceutical education.

[49]  A. Hafdahl,et al.  Combining Heterogeneous Correlation Matrices: Simulation Analysis of Fixed-Effects Methods , 2008 .

[50]  Michael Schreiber.  An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the A-index, and the R-index , 2008 .

[51]  Mônica G. Campiteli,et al.  Is it possible to compare researchers with different scientific interests? , 2006, Scientometrics.

[52]  Larry V. Hedges,et al.  Fixed-Effects Models , 2022, The SAGE Encyclopedia of Research Design.

[53]  Wolfgang Glänzel,et al.  On the Opportunities and Limitations of the H-index , 2006 .

[54]  Steven D. Johnson,et al.  Assessment of Research Performance in Biology: How Well Do Peer Review and Bibliometry Correlate , 2008 .

[55]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[56]  Leo Egghe,et al.  The Hirsch index and related impact measures , 2010, Annu. Rev. Inf. Sci. Technol..

[57]  Michael Schreiber,et al.  A case study of the Hirsch index for 26 non‐prominent physicists , 2007, 0708.0120.

[58]  Leo Egghe,et al.  An h-index weighted by citation impact , 2008, Inf. Process. Manag..

[59]  Mônica G. Campiteli,et al.  An index to quantify an individual's scientific research valid across disciplines , 2005 .

[60]  F. J. Cabrerizoa,et al.  q 2-Index : Quantitative and qualitative evaluation based on the number and impact of papers in the Hirsch core , 2009 .

[61]  L. Egghe.  An improvement of the h-index: the g-index , 2006 .

[62]  W. Couldwell,et al.  Use of the h index in neurosurgery. Clinical article. , 2009, Journal of neurosurgery.

[63]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[64]  A. Field.  Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. , 2001, Psychological methods.

[65]  Miguel A García-Pérez,et al.  The Hirsch h Index in a Non-Mainstream Area: Methodology of the Behavioral Sciences in Spain , 2009, The Spanish journal of psychology.

[66]  David Navon,et al.  The h-i index: A proposed new metric of individual scientific output , 2009 .

[67]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[68]  Ronald Rousseau,et al.  A discussion of Prathap's h2-index for institutional evaluation with an application in the field of HIV infection and therapy , 2010, J. Informetrics.

[69]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[70]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[71]  Richard S. J. Tol,et al.  The h-index and its alternatives: An application to the 100 most prolific economists , 2009, Scientometrics.

[72]  Rodrigo Costas,et al.  Is g-index better than h-index? An exploratory study at the individual level , 2008, Scientometrics.

[73]  Ludo Waltman,et al.  Generalizing the H- and G-Indices , 2008, J. Informetrics.

[74]  M. Kosmulski.  A new Hirsch-type index saves time and works equally well as the original h-index , 2009 .

[75]  Robin K. S. Hankin,et al.  Beyond the Durfee square: Enhancing the h-index to score total publication output , 2008, Scientometrics.

[76]  Lutz Bornmann,et al.  Hirsch‐Type Index Values for Organic Chemistry Journals: A Comparison of New Metrics with the Journal Impact Factor , 2009 .

[77]  Jean-François Molinari,et al.  A new methodology for ranking scientific institutions , 2008, Scientometrics.

[78]  Yang Li-ying,et al.  Bibliometrical Analysis of Paper's Co-occurrence in Main International Institutions&mdash|A Case Study in Chemical Research , 2006 .

[79]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[80]  Marek Kosmulski,et al.  MAXPROD - A New Index for assessment of the scientific output of an individual, and a comparison with the h-index , 2007 .

[81]  J. Hox.  Multilevel analysis: Techniques and applications, 2nd ed. , 2010 .

[82]  Theo Stijnen,et al.  Advanced methods in meta‐analysis: multivariate approach and meta‐regression , 2002, Statistics in medicine.

[83]  Ronald Rousseau,et al.  Influence of individual researchers’ visibility on institutional impact: an example of Prathap’s approach to successive h-indices , 2009, Scientometrics.

[84]  Diane M. Tomasic,et al.  Meta-Analysis: A Comparison of Approaches , 2005 .

[85]  S. Wulff.  SAS for Mixed Models , 2007 .

[86]  Ronald Rousseau,et al.  In those fields where multiple authorship is the rule, the h-index should be supplemented by role-based h-indices , 2010, J. Inf. Sci..

[87]  András Schubert,et al.  Hirsch-type indices for characterizing networks , 2009, Scientometrics.

[88]  Angelos Hatzakis,et al.  Assessing the impact of biomedical research in academic institutions of disparate sizes , 2009, BMC medical research methodology.

[89]  L. Bornmann,et al.  The state of h index research , 2009, EMBO reports.

[90]  A A M Wilde,et al.  The Hirsch-index: a simple, new tool for the assessment of scientific output of individual scientists , 2009, Netherlands heart journal : monthly journal of the Netherlands Society of Cardiology and the Netherlands Heart Foundation.

[91]  András Schubert,et al.  Successive h-indices , 2007, Scientometrics.

[92]  Michael Schreiber,et al.  An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the A-index, and the R-index , 2008, J. Assoc. Inf. Sci. Technol..

[93]  Francisco Herrera,et al.  hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices , 2010, Scientometrics.

[94]  Harvey Goldstein,et al.  Multilevel Cross-Classified Models , 1994 .

[95]  Claes Wohlin,et al.  A new index for the citation curve of researchers , 2009, Scientometrics.

[96]  Mourad Touzani,et al.  Ranking marketing journals using the Google Scholar-based hg-index , 2010, J. Informetrics.

[97]  Qiang Wu,et al.  The w-index: A measure to assess scientific impact by focusing on widely cited papers , 2010, J. Assoc. Inf. Sci. Technol..

[98]  Ping-huan Hua,et al.  A Perfect Hirsch-type Index? Experiences Using the Tapered h-Index (hT) , 2010 .

[99]  Quentin L. Burrell.  Some comments on "A Proposal for a Dynamic h-Type Index" by Rousseau and Ye , 2009, J. Assoc. Inf. Sci. Technol..

[100]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[101]  H. D. White.  On Extending Informetrics: An Opinion Paper , 2008 .

[102]  John Mingers,et al.  Measuring the research contribution of management academics using the Hirsch-index , 2009, J. Oper. Res. Soc..

[103]  N. Haslam,et al.  Quality, quantity, and impact in academic publication , 2009 .

[104]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[105]  Phillip I. Good,et al.  Common Errors in Statistics (and How to Avoid Them) , 2003 .

[106]  Peter Vinkler,et al.  The πv-index: a new indicator to characterize the impact of journals , 2010, Scientometrics.

[107]  Adam R Hafdahl,et al.  Meta-analysis of correlations revisited: attempted replication and extension of Field's (2001) simulation studies. , 2009, Psychological methods.

[108]  Lokman I. Meho,et al.  A proposal for a dynamic h-type index , 2008 .

[109]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[110]  Miguel A. García-Pérez,et al.  A multidimensional extension to Hirsch’s h-index , 2009, Scientometrics.