The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach

When comparing the citation impact of nations, departments or other groups of researchers within individual fields, three approaches have been proposed: arithmetic means, geometric means, and percentage in the top X%. This article compares the precision of these statistics using 97 trillion experimentally simulated citation counts from 6875 sets of different parameters (although all having the same scale parameter) based upon the discretised lognormal distribution with limits from 1000 repetitions for each parameter set. The results show that the geometric mean is the most precise, closely followed by the percentage of a country's articles in the top 50% most cited articles for a field, year and document type. Thus the geometric mean citation count is recommended for future citation-based comparisons between nations. The percentage of a country's articles in the top 1% most cited is a particularly imprecise indicator and is not recommended for international comparisons based on individual fields. Moreover, whereas standard confidence interval formulae for the geometric mean appear to be accurate, confidence interval formulae are less accurate and consistent for percentile indicators. These recommendations assume that the scale parameters of the samples are the same but the choice of indicator is complex and partly conceptual if they are not.

[1]  T. S. Evans,et al.  Universality of performance indicators based on citation and reference counts , 2011, Scientometrics.

[2]  Giovanni Abramo,et al.  The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy , 2015, J. Informetrics.

[3]  A. D. Moivre.  The Doctrine of Chances: Or a Method of Calculating the Probability of Events in Play , 2008 .

[4]  Bikas K. Chakrabarti,et al.  Universality of Citation Distributions for Academic Institutions and Journals , 2014, PloS one.

[5]  Mike Thelwall,et al.  Distributions for cited articles from individual subjects and years , 2014, J. Informetrics.

[6]  Antonio Perianes-Rodríguez,et al.  Differences in citation impact across countries , 2015, J. Assoc. Inf. Sci. Technol..

[7]  Mark E. J. Newman,et al.  Power-Law Distributions in Empirical Data , 2007, SIAM Rev..

[8]  Derek de Solla Price,et al.  A general theory of bibliometric and other cumulative advantage processes , 1976, J. Am. Soc. Inf. Sci..

[9]  Mike Thelwall,et al.  Which factors help authors produce the highest impact research? Collaboration, journal and document properties , 2013, J. Informetrics.

[10]  Vincent Larivière,et al.  Tweets vs. Mendeley readers: How do these two social media metrics differ? , 2014, it Inf. Technol..

[11]  J. S. Hunter,et al.  Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. , 1979 .

[12]  Loet Leydesdorff,et al.  Remaining problems with the "New Crown Indicator" (MNCS) of the CWTS , 2010, J. Informetrics.

[13]  Xiao-Hua Zhou,et al.  Confidence intervals for the log-normal mean . , 1997, Statistics in medicine.

[14]  A. F. J. VAN RAAN,et al.  In matters of quantitative studies of science the fault of theorists is offering too little and asking too much , 1998, Scientometrics.

[15]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[16]  Ronald Rousseau,et al.  Median and percentile impact factors: A set of new indicators , 2005, Scientometrics.

[17]  W. Stahel,et al.  Log-normal Distributions across the Sciences: Keys and Clues , 2001 .

[18]  Philip J. Fleming,et al.  How not to lie with statistics: the correct way to summarize benchmark results , 1986, CACM.

[19]  Thelwall Mike.  R code and results for: The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach , 2015 .

[20]  D. King.  The scientific impact of nations , 2004, Nature.

[21]  J. Hilbe.  Negative Binomial Regression: Preface , 2007 .

[22]  Leslie D. Hall,et al.  Science and Engineering Indicators 2010. NSB 10-01. , 2010 .

[23]  Michael Schreiber,et al.  How much do different ways of calculating percentiles influence the derived performance indicators? A case study , 2013, Scientometrics.

[24]  Michel Zitt,et al.  The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay’s article 2011 , 2012, Scientometrics.

[25]  Tibor Braun,et al.  Cross-field normalization of scientometric indicators , 1996, Scientometrics.

[26]  Jesper W. Schneider,et al.  Caveats for using statistical significance tests in research assessments , 2011, J. Informetrics.

[27]  Jonas Lundberg,et al.  Lifting the crown - citation z-score , 2007, J. Informetrics.

[28]  Antonio Perianes-Rodríguez,et al.  University citation distributions , 2015, ISSI.

[29]  Loet Leydesdorff,et al.  The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits , 2012, J. Informetrics.

[30]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.

[31]  P. Seglen,et al.  Citation rates and journal impact factors are not suitable for evaluation of research. , 1998, Acta orthopaedica Scandinavica.

[32]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1992, Scientometrics.

[33]  Mike Thelwall,et al.  National, disciplinary and temporal variations in the extent to which articles with more authors have more impact: Evidence from a geometric field normalised citation indicator , 2015, J. Informetrics.

[34]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[35]  Michal Brzezinski,et al.  Power laws in citation distributions: evidence from Scopus , 2014, Scientometrics.

[36]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[37]  Dag W. Aksnes,et al.  Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods , 2012, J. Informetrics.

[38]  Santo Fortunato,et al.  Characterizing and Modeling Citation Dynamics , 2011, PloS one.

[39]  Colin S Gillespie,et al.  Fitting Heavy Tailed Distributions: The poweRlaw Package , 2014, 1407.3492.

[40]  Chaomei Chen,et al.  Predictive effects of structural variation on citation counts , 2012, J. Assoc. Inf. Sci. Technol..

[41]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[42]  Giovanni Abramo,et al.  The VQR, Italy's second national research assessment: Methodological failures and ranking distortions , 2015, J. Assoc. Inf. Sci. Technol..

[43]  Mike Thelwall,et al.  More precise methods for national research citation impact comparisons , 2015, J. Informetrics.

[44]  Mike Thelwall,et al.  National research impact indicators from Mendeley readers , 2015, J. Informetrics.

[45]  C. Land,et al.  An Evaluation of Approximate Confidence Interval Estimation Methods for Lognormal Means , 1972 .

[46]  Ludo Waltman,et al.  On the calculation of percentile-based bibliometric indicators , 2012, J. Assoc. Inf. Sci. Technol..

[47]  Industrial Strategy,et al.  International comparative performance of the UK research base , 2012 .

[48]  Mike Thelwall,et al.  Regression for citation data: An evaluation of different methods , 2014, J. Informetrics.

[49]  Amy M. Hightower,et al.  Science and Engineering Indicators , 1993 .

[50]  Søren Asmussen,et al.  Asymptotics of sums of lognormal random variables with Gaussian copula , 2008 .

[51]  Mike Thelwall,et al.  Stopped Sum Models for Citation Data , 2015, ISSI.