Uncertainty Measures for Economics Journal Impact Factors

Academic economists appear to be intensely interested in rankings of journals, institutions, and individuals. Yet there is little discussion of the uncertainty associated with these rankings. To illustrate the uncertainty associated with citations-based rankings, I compute the standard error of the impact factor for all economics journals with a five-year impact factor in the 2011 Journal Citations Report. I use these to derive confidence intervals for the impact factors as well as ranges of possible rank for a subset of thirty journals. I find that the impact factors of the top two journals are well defined and set these journals apart in a clearly defined group. An elite group of 9-11 mainstream journals can also be fairly reliably distinguished. The four bottom ranked journals are also fairly clearly set apart. For the remainder of the distribution, confidence intervals overlap and rankings are quite uncertain.

[1]  Wolfgang Glänzel.  The multi-dimensionality of journal impact , 2008, Scientometrics.

[2]  John Hudson,et al.  Be known by the company you keep: Citations — quality or chance? , 2007, Scientometrics.

[3]  Gaines H. Liner.  Core Journals in Economics , 2002 .

[4]  Hui-yun Shen,et al.  The outflow of academic papers from China: why is it happening and can it be stemmed? , 2011 .

[5]  Michael Burton,et al.  Core journals: A reappraisal of the Diamond list , 1995 .

[6]  Pantelis Kalaitzidakis,et al.  RANKINGS OF ACADEMIC JOURNALS AND INSTITUTIONS IN ECONOMICS , 2003 .

[7]  A. Oswald.  An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers , 2006, SSRN Electronic Journal.

[8]  U. Fischbacher.  z-Tree: Zurich toolbox for ready-made economic experiments , 1999 .

[9]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[10]  S. Liebowitz,et al.  Assessing the Relative Impacts of Economic Journals , 1984 .

[11]  Howard J. Wall,et al.  Don't Get Skewed Over by Journal Rankings , 2009 .

[12]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[13]  Henk F. Moed,et al.  Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way , 2012, Scientometrics.

[14]  David J. Olive.  A Simple Confidence Interval for the Median , 2005 .

[15]  Anthony F. J. van Raan,et al.  Properties of journal impact in relation to bibliometric research group performance indicators , 2012, Scientometrics.

[16]  Steve Pressé,et al.  Nonuniversal power law scaling in the probability distribution of scientific citations , 2010, Proceedings of the National Academy of Sciences.

[17]  T. Opthof,et al.  Sense and nonsense about the impact factor. , 1997, Cardiovascular research.

[18]  David I. Stern,et al.  Environmental and Ecological Economics: A Citation Analysis , 2006 .

[19]  Christian Zimmermann,et al.  Academic Rankings with RePEc , 2012 .

[20]  Gabriel Pinski,et al.  Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics , 1976, Inf. Process. Manag..

[21]  Jufang Shao,et al.  The outflow of academic papers from China: why is it happening and can it be stemmed? , 2011, Learn. Publ..

[22]  D. Greenwood,et al.  Selective attrition and bias in a longitudinal health survey among survivors of a disaster , 2007, BMC medical research methodology.

[23]  D. Laband,et al.  The Relative Impacts of Economics Journals: 1970-1990 , 1994 .

[24]  P. Hegarty,et al.  The Consequences of Predicting Scientific Impact in Psychology Using Journal Impact Factors , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[25]  Carl T. Bergstrom.  Eigenfactor Measuring the value and prestige of scholarly journals , 2007 .

[26]  Oscar Volij,et al.  The Measurement of Intellectual Influence , 2002 .

[27]  Alexander I. Pudovkin,et al.  Rank normalization of impact factors will resolve Vanclay’s dilemma with TRIF , 2012, Scientometrics.

[28]  T. Coupé.  REVEALED PERFORMANCES: WORLDWIDE RANKINGS OF ECONOMISTS AND ECONOMICS DEPARTMENTS, 1990–2000 , 2003 .

[29]  George Emm. Halkos,et al.  Measuring economic journals’ citation efficiency: a data envelopment analysis approach , 2011, Scientometrics.

[30]  S. Redner.  How popular is your paper? An empirical study of the citation distribution , 1998, cond-mat/9804163.

[31]  Jerome K. Vanclay,et al.  Impact factor: outdated artefact or stepping-stone to journal certification? , 2011, Scientometrics.

[32]  David N. Laband,et al.  The determinants of article popularity: Preliminary results , 1985 .

[33]  Joseph L. Gastwirth,et al.  On a Simple Estimate of the Reciprocal of the Density Function , 1968 .

[34]  M. Sales-Pardo,et al.  Effectiveness of Journal Ranking Schemes as a Tool for Locating Information , 2008, PloS one.

[35]  Johan Bollen,et al.  A Principal Component Analysis of 39 Scientific Impact Measures , 2009, PloS one.

[36]  Wolfgang Glänzel,et al.  Statistical reliability of comparisons based on the citation impact of scientific publications , 2005, Scientometrics.

[37]  Mario De Marchi,et al.  The editorial policies of scientific journals: Testing an impact factor model , 2001, Scientometrics.

[38]  Robert D. Tollison,et al.  Dry Holes in Economic Research , 2003 .

[39]  Glenn Ellison.  The Slowdown of the Economics Publishing Process , 2000 .

[40]  Amber E. Budden,et al.  Bang for Your Buck: Rejection Rates and Impact Factors in Ecological Journals , 2008 .

[41]  Vincent Larivière,et al.  The weakening relationship between the impact factor and papers' citations in the digital age , 2012, J. Assoc. Inf. Sci. Technol..

[42]  R. Dusansky,et al.  Rankings of U.S. Economics Departments , 1998 .