A national research profile-based immediacy index and citation ratio indicator for research evaluation

Introduction In this study we analyzed the behaviour of Fieldbased Crown Indicators (here named FCI) for two smaller EU countries, Denmark and Finland, as well as for the European Union and globally calculated over three periods, 2001-2005, 2004 (cited 2004-05) and the most recent year, 2005. The purpose was to observe the robustness and stability of the indicator as a National Immediacy Index during the latter two periods compared to the five-year window. The motivation is to provide national research funding agencies with information on the relative citation impact and national citation visibility of the most recent output in central research areas. To this end FCI (Moed et al., 1995) is applied based on data from the National Science Indicators database (NSI, Thomson Scientific). It is calculated based on the publication profile for each country, as defined by 19 broader scientific fields. The citation visibility is defined as the ratio of received citations by the country over the number of expected citations. With respect to the citation visibility two calculations are done: First, for each field its global citation impact is multiplied by its national publication volume, providing the number of expected citations for that field. Essentially, this value represents a ‘shadow-field’ of the original national research field. When compared to the actual number of citations obtained by that field it is possible to observe the number of citations gained or lost. Secondly, the aggregated values over all the ‘shadow-fields’ signify the expected citation volume of the country in question. Finally, when calculating the mean citation visibility ratio the underlying national publication shares of the research fields (here 19) are acting as weights. The present study assumes that the recent five-year citation and publication window, 2001-2005, provides a robust set of indicators. However, one would like to obtain assessment information on research as recent as possible – in the form of a kind of National Immediacy Index (NII). We compared the citation performance during the three snapshots by diachronic citation analyses. The aim was to observe if 1) it makes sense to apply broader research fields, but in very short and recent time frames (robustness); 2) there exist a significant correlation between impact and visibility results obtained from the three snapshots (stability). Humanity fields were excluded but social science fields were included as one of the 19 central fields forming the profile taken from NSI (2006). Publication types covered journal and review articles, notes and proceedings papers from journals and selected central research conferences. The ‘Multidisciplinary’ field in NSI adheres mainly to the sciences but does not contain field-specific articles from Nature, PNAS, Science, etc., which are placed in their proper fields. The Public Health subfield was included in Clinical Medicine (extracted from Social Science, general). The NSI feature “oneyear period cited-to-present” was used for the data collection. National self-citations are included. In order to be robust we considered that each field in a NII must demonstrate at least 75 publications as a threshold for the most recent year, corresponding roughly to 15 citations received in a central field over the most recent year, with an impact of .20.