Estimating the accuracies of journal impact factor through bootstrap

The journal impact factor (JIF) reported in journal citation reports has been used to represent the influence and prestige of a journal. Whereas the consideration of the stochastic nature of a statistic is a prerequisite for statistical inference, the estimation of JIF uncertainty is necessary yet unavailable for comparing the impact among journals. Using journals in the Database of Research in Science Education (DoRISE), the current study proposes bootstrap methods to estimate the JIF variability. The paper also provides a comprehensive exposition of the sources of JIF variability. The collections of articles in the year of interest and in the preceding years both contribute to JIF variability. In addition, the variability estimate differs depending on the way a database selects its journals for inclusion. In the bootstrap process, the nested structure of articles in a journal was accounted for to ensure that each bootstrap replication reflects the actual citation characteristics of articles in the journal. In conclusion, the proposed point and interval estimates of the JIF statistic are obtained and more informative inferences on the impact of journals can be drawn.

[1]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[2]  Richard M. Smith,et al.  Commentary: the power of the unrelenting impact factor--is it a force for good or harm? , 2006, International journal of epidemiology.

[3]  Giovanni Abramo,et al.  Citations versus journal impact factor as proxy of quality: could the latter ever be preferable? , 2010, Scientometrics.

[4]  Loet Leydesdorff.  Does the specification of uncertainty hurt the progress of scientometrics? , 2013, J. Informetrics.

[5]  Per Ottar Seglen,et al.  The skewness of science , 1992 .

[6]  Pablo Dorta-González,et al.  Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor , 2012, Scientometrics.

[7]  Werner Marx,et al.  The impact of Solid State Communications in view of the ISI Citation data , 2003 .

[8]  Bradley Efron,et al.  Censored Data and the Bootstrap , 1981 .

[9]  H. Small,et al.  Modifying the journal impact factor by fractional citation weighting: The audience factor , 2008 .

[10]  A Donner,et al.  Methods for comparing event rates in intervention studies when the unit of allocation is a cluster. , 1994, American journal of epidemiology.

[11]  Jesper W. Schneider,et al.  Caveats for using statistical significance tests in research assessments , 2011, J. Informetrics.

[12]  A. Paraskeva,et al.  Self-citations in six anaesthesia journals and their significance in determining the impact factor. , 2000, British journal of anaesthesia.

[13]  Irina Marshakova-Shaikevich,et al.  The standard impact factor as an evaluation tool of science fields and scientific journals , 1996, Scientometrics.

[14]  Pandelis Perakakis,et al.  Measuring internationality: Reflections and perspectives on academic journals , 2006, Scientometrics.

[15]  Brink Aj,et al.  Impact factor: use and abuse. , 2004, Cardiovascular journal of South Africa : official journal for Southern Africa Cardiac Society [and] South African Society of Cardiac Practitioners.

[16]  L. Leydesdorff,et al.  Dimensions of Citation Analysis , 1990 .

[17]  O. Pons.  Bootstrap of means under stratified sampling , 2007, 0709.3246.

[18]  E. Garfield.  Journal impact factor: a brief review. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[19]  Narongrit Sombatsompop,et al.  Making an equality of ISI impact factors for different subject fields , 2005, J. Assoc. Inf. Sci. Technol..

[20]  D. Greenwood,et al.  Selective attrition and bias in a longitudinal health survey among survivors of a disaster , 2007, BMC medical research methodology.

[21]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[22]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[24]  A A Sandberg,et al.  The journal "impact factor": a misnamed, misleading, misused measure. , 1998, Cancer genetics and cytogenetics.

[25]  Morris H. Hansen,et al.  Sample survey methods and theory , 1955 .

[26]  Daryl E. Chubin,et al.  Is citation analysis a legitimate evaluation tool? , 1979, Scientometrics.

[27]  Thed N. van Leeuwen,et al.  Some modifications to the SNIP journal impact indicator , 2012, J. Informetrics.

[28]  E. Garfield.  Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[29]  Tze Leung Lai,et al.  BOOTSTRAP METHODS FOR TRUNCATED AND CENSORED DATA , 1996 .

[30]  Lucio Barabesi,et al.  Statistical inference on the h-index with an application to top-scientist performance , 2012, J. Informetrics.

[31]  Thed N. van Leeuwen,et al.  Development and application of journal impact measures in the Dutch science system , 2002, Scientometrics.

[32]  Sharon L. Lohr,et al.  Sampling: Design and Analysis , 1999 .

[33]  Frederic M. Lord,et al.  A strong true-score theory, with applications , 1965 .

[34]  Ana María Ramírez,et al.  Renormalized Impact Factor , 2004, Scientometrics.

[35]  B. Efron.  © Institute of Mathematical Statistics, 2003 Second Thoughts on the Bootstrap , 2022 .

[36]  Wolfgang Glänzel,et al.  Statistical reliability of comparisons based on the citation impact of scientific publications , 2005, Scientometrics.

[37]  Patrick Onghena,et al.  An Assessment of the Predictive Validity of Impact Factor Scores: Implications for Academic Employment Decisions in Social Work , 2006 .