On the meaningful and non-meaningful use of reference sets in bibliometrics

In a paper published recently, Kaur, Radicchi, and Menczer (2013) used data from the Scholarometer http://scholarometer.indiana.edu/) to examine the effectiveness of various metrics, such as the h index (Hirsch, 2005) and he new crown indicator (Lundberg, 2007; Opthof & Leydesdorff, 2010; Waltman, van Eck, van Leeuwen, Visser, & van Raan, 011a; Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011b) in generating field-normalized citation scores. While he subject of field-normalization in bibliometrics has up to now been discussed primarily at the paper level (Bornmann Leydesdorff, 2013), Kaur et al. (2013) have been looking at the author level: which metric allows an effective method of eld-normalization with which scientists can be compared across different fields? There have already been numerous calls or benchmarks for comparative assessment in order to evaluate individual scientists (Garfield, 1979; Kreiman & Maunsell, 011). Something that is very difficult to implement for individuals (Coleman, Bolumole, & Frankel, 2012; El Emam, Arbuckle, onker, & Anderson, 2012) can be achieved at the institutional level with the data from the rankings, as Bornmann and de oya Anegón (2014) have shown in their study (provided that one accepts the reasoned decisions which are made for onstructing the rankings). We would like to use the paper by Kaur et al. (2013) as a starting point for an investigation into an appropriate method f field-normalization (at individual scientist level). In bibliometrics, in order to allow cross-disciplinary comparisons of itation impact at the level of individual papers, a reference set made up of all the papers from the same field (and the same ublication year) is compiled for each paper. One can expect that the Web of Science (WoS, Thomson Reuters) and Scopus Elsevier) have a good coverage of the literature in the natural and life sciences (Mahdi, d’Este, & Neely, 2008). Only by taking ll the comparable papers into account is it possible to ensure that the measurement of the impact of the paper in question s valid compared to similar papers (Bornmann & Marx, 2013a). If only some of the total numbers of papers are used in the eference sets, there is no comparison with the relevant reference sets. In their study, Kaur et al. (2013) use bibliometric data for scientists who have used the Scholarometer tool to normalize etrics at the level of individual scientists. As we can assume that not all scientists worldwide use this tool (nor a sample hich can be interpreted as meaningful, such as all scientists with at least one paper in WoS), it is not possible to generate eaningful and valid reference data at the level of scientists on this basis. In order to test the various metrics in their study, he authors would have had to normalize each scientist recorded in Scholarometer with an appropriate valid reference set including all scientists). Only then would they have been able to verify the effectiveness of the metrics using the example f scientists from Scholarometer. Where the new crown indicator is concerned, which Kaur et al. (2013) included in their comparative study along with thers, there is an additional problem in that it was obviously also calculated with Scholarometer data. The expected citation ate for a publication is therefore not – as is a currently standard in bibliometrics – calculated over the impact of all the ublications in a subject category of the Web of Science (WoS, Thomson Reuters) or in a Scopus (Elsevier) subject area and publication year, but over the impact of a selection of publications which users of the Scholarometer have entered quite andomly and assigned to certain disciplines. Can we call this a valid reference set? We would say not. Bibliometric data used to evaluate research on the level of individual scientists is highly critical data and should therefore e compiled very carefully (Bornmann & Marx, 2013b; Marx & Bornmann, 2014). In the Scholarometer, users can enter names f scientists as they wish, assign these scientists to certain disciplines, and compile their publication sets. Even though there re processes implemented in the Scholarometer which are supposed to prevent serious misclassifications (Kaur et al., 2012), ne can assume that these assignments are not high-quality. However, we need high-quality data when we put reference ets together and use them for the evaluation of research. Indeed, the database operator Chemical Abstracts Services, for xample, employs a number of highly specialised people to assign individual publications in chemistry and its related fields o specific subject categories (Bornmann, Marx, & Barth, 2013; Bornmann, Mutz, Marx, Schier, & Daniel, 2011). Thomson euters (WoS) and Elsevier (Scopus) have addressed the problem of subject matter classification by assigning journals (and ot individual papers) to subject categories. Despite much criticism (Bornmann, Mutz, Neuhaus, & Daniel, 2008b; Rafols & eydesdorff, 2009), this classification is currently used as a standard in bibliometrics.