Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization

Since many years the so called crown indicator (CPP/FCSm) for field normalization of the Centre for Science and Technolgy Studies (CWTS, Leiden, The Netherlands) has been defined as being the standard in the evaluative bibliometric practice n many different contexts. The publication of the paper by Opthof and Leydesdorff (2010) was a starting point in the field of valuative bibliometrics to challenge this CWTS standard indicator for evaluative purposes. Meanwhile, the paper of Opthof nd Leydesdorff (2010) has been extensively discussed (Van Raan, Van Leeuwen, Visser, Van Eck, & Waltman, 2010), e.g., uring the Science and Technology Indicators conference 2010 in Leiden (Anon, 2010), and a new crown indicator was preented by CWTS: the mean normalized citation score (MNCS) (Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011). At ts heart both, the previous and new indicator differentiates as follows: whereas for the previous crown indicator the average itation rate over all papers in a publication set is calculated and then the citation rate is field-normalized, for the new crown ndicator each paper’s citation impact in a paper set becomes field-normalized, before an average value (harmonic average) ver the field-normalized impact values is calculated. Not only for the previous but also for the new crown indicator the disadvantage of resting on the arithmetic average xists (Leydesdorff & Opthof, in press): the mean citation impact values calculated for different fields are arithmetic averages nd the crown indicators rest on arithmetic averages of ratios or ratios of arithmetic averages. A number of publications has ointed out that in the face of non-normal distributed citation data, the arithmetic mean value is not appropriate as a measure f central tendency. It can give a distorted picture of the kind of distribution (Bornmann, Mutz, Neuhaus, & Daniel, 2008), and it is a rather crude statistic” (Joint Committee on Quantitative Assessment of Research, 2008, p. 2). As the distribution of itation counts is usually right-skewed, distributed according to a power law (Joint Committee on Quantitative Assessment f Research, 2008), arithmetic average citation rates mainly show where papers with high citation counts are to be found. ccording to Evidence Ltd (2007)“where bibliometric data must stand-alone, they should be treated as distributions and ot as averages” (p. 10). What is more, the calculation of ratios runs into serious problems regarding the interpretation of itation impact as low or high (or excellent). The interpretation is more or less arbitrary without using reference distributions. ince many years, the evaluation of the citation performance of research groups as far below (<0.5) or far above (>1.5) the nternational citation impact standard has been based on cut-off points developed by personal experiences (see here van aan, 2005). In the following we will present an extension of the proposals of Bornmann (2010) for an improved practise of fieldormalized citation performance measurement. The extension is intended to calculate a single measure for the citation mpact of a group of scientists that is not based on the arithmetic average but uses reference distributions. The measure llows – similar to the previous and new crown indicator – to compare groups of scientists by using one single number. he proposals of Bornmann (2010) are based on the calculation of percentiles. The use of percentiles for citation data is ery advantageous, because no assumptions have to be made as to the underlying distribution of citations (Boyack, 2004). ith percentiles each paper in a paper set of a research group can be field-normalized with a matching reference standard. o generate the reference standard for a paper in question all papers published in the same year, with the same document ype and belonging to the same field (defined by a discipline-oriented database) are categorized into six percentile impact lasses: 99th – top 1%, 95th, 90th, 75th, 50th, and <50th – bottom 50% (following the approach of the National Science Board, 010) (see here also Bornmann, de Moya-Anegón, & Leydesdorff, 2010). Through the use of the citation limits that define hese classes the paper in question can be assigned to one of the six citation impact classes. This procedure is repeated for ll papers published by a research group (The use of the percentile for each paper instead of the corresponding percentile mpact class might be more preferable (e.g., resulting in a higher power), but this is a very cumbersome and expensive task or a bigger publication set.). Bornmann (2010) presents the results of an evaluative citation analysis calculated with fictitious bibliometric data for hree research groups. Table 1 shows this data with some additional numbers. As the table reveals the papers of the groups ere categorized into six percentile impact classes (99th – top 1%, 95th, 90th, 75th, 50th, and <50th – bottom 50%). First of all,