On the stability of citation-based journal rankings

The article presents a large-scale comparison of journal rankings based on seven impact measures: Impact Factor (2- and 5-year), SJR, IPP, SNIP, H index, and Article Influence Score. Three aspects of ranking stability in the 2007–2014 period were analyzed: temporal, cross-discipline, and cross-indicator. Impact measures based on five-year citation windows enable more stable journal rankings over time. Journal rankings based on the source-normalized indicator (SNIP) have the largest cross-discipline stability. Journals in the fields of social sciences and humanities have lower temporal and cross-discipline ranking stability compared to those in “hard” sciences. Although correlation coefficients indicate relatively high agreement among the rankings based on different indicators, variations in quartile and percentile ranks suggest different conclusions. WoS journals almost linearly improve their ranking positions in Scopus lists, while many high-impact journals covered by Scopus are not available in WoS. An important element of the ranking stability is the discriminability of impact measures. Beyond the segregation between the top and bottom ranked journals, our assessment of “quality” relies in most cases on a rather arguable assumption that a couple of citations more or less is making a big difference.

[1]  Mark D. Uncles,et al.  Commentary on the Mort et al. Paper: Journal Rankings: How Much Credence Should We Give Them? , 2004 .

[2]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics. , 2015, Nature.

[3]  Ismael Rafols,et al.  How journal rankings can suppress interdisciplinary research: A comparison between Innovation Stud , 2012 .

[4]  V. Beattie,et al.  A new method for ranking academic journals in accounting and finance , 2005 .

[5]  Vincent Larivière,et al.  The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities , 2006, J. Assoc. Inf. Sci. Technol..

[6]  M. Brysbaert,et al.  Journal impact factors and self-citations: implications for psychology journals. , 2004, The American psychologist.

[7]  Philip M. Davis.  Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts? , 2008 .

[8]  Konrad Kulakowski,et al.  On the quality evaluation of scientific entities in Poland supported by consistency-driven pairwise comparisons method , 2014, Scientometrics.

[9]  Dalibor Fiala,et al.  Science Evaluation in the Czech Republic: The Case of Universities , 2013 .

[10]  Juan Miguel Campanario,et al.  Large increases and decreases in journal impact factors in only one year: The effect of journal self-citations , 2011, J. Assoc. Inf. Sci. Technol..

[11]  Enrique Orduña-Malea,et al.  Google Scholar Metrics evolution: an analysis according to languages , 2013, Scientometrics.

[12]  Mathieu Bastian,et al.  Gephi: An Open Source Software for Exploring and Manipulating Networks , 2009, ICWSM.

[13]  Bart De Moor,et al.  Hybrid clustering for validation and improvement of subject-classification schemes , 2009, Inf. Process. Manag..

[14]  Henk F. Moed,et al.  New developments in the use of citation analysis in research evaluation , 2009, Archivum Immunologiae et Therapiae Experimentalis.

[15]  Ronald Rousseau,et al.  A quick and easy method to estimate the random effect on citation measures , 2005, Scientometrics.

[16]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[17]  Donald S. Siegel,et al.  A Theoretical and Empirical Analysis of Journal Rankings: The Case of Formal Lists , 2000 .

[18]  Vicente P. Guerrero-Bote,et al.  A new approach to the metric of journals' scientific prestige: The SJR indicator , 2010, J. Informetrics.

[19]  Pablo Dorta-González,et al.  Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor , 2012, Scientometrics.

[20]  Aliakbar Haghdoost,et al.  How variable are the journal impact measures? , 2014, Online Inf. Rev..

[21]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[22]  Juan Gorraiz,et al.  On the temporal stability of Garfield’s Impact Factor and its suitability to identify hot papers , 2012 .

[23]  M Sutter,et al.  Tools for Evaluating Research Output , 2001, Evaluation review.

[24]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[25]  S. Black.  How Much Do Core Journals Change over a Decade , 2012 .

[26]  A. Paraskeva,et al.  Impact factor bias and proposed adjustments for its determination , 2002, Acta anaesthesiologica Scandinavica.

[27]  M. Segalla.  Publishing in the right place or publishing the right thing: journal targeting and citations' strategies for promotion and tenure committees , 2008 .

[28]  Ju-fang Shao,et al.  Research assessment and monetary rewards: the overemphasized impact factor in China , 2012 .

[29]  Jüri Allik,et al.  FACTORS AFFECTING BIBLIOMETRIC INDICATORS OF SCIENTIFIC QUALITY , 2012 .

[30]  D. Pontille,et al.  The controversial policies of journal ratings: evaluating social sciences and humanities , 2010 .

[31]  Diana Hicks,et al.  One size doesn't fit all: On the co-evolution of national evaluation systems and social science publishing , 2012 .

[32]  Julie Pallant,et al.  SPSS survival manual : a step by step guide to data analysis using SPSS for Windows , 2001, Behaviour Change.

[33]  Ronald Rousseau,et al.  Journal Evaluation: Technical and Practical Issues, , 2002, Libr. Trends.

[34]  E. Russo,et al.  SEARCH CONTINUES FOR BIOCHEMICAL PAHTWAY THAT LEADS TO ONSET OF ALZHEIMER'S DISEASE , 1998 .

[35]  R. Cagan.  The San Francisco Declaration on Research Assessment , 2013, Disease Models & Mechanisms.

[36]  E. Picard.  San Francisco declaration on research assessment , 2013 .

[37]  E. Todeva,et al.  Accounting for Research Quality: Research Audits and the Journal Rankings Debate , 2015 .

[38]  Tanja Jevremov,et al.  Globally national - locally international: Bibliometric analysis of a SEE psychology journal , 2014 .

[39]  Carl T. Bergstrom,et al.  Differences in impact factor across fields and over time , 2009 .

[40]  Christopher B. Barrett,et al.  Subdiscipline-Specific Journal Rankings: Whither Applied Economics? , 1998 .

[41]  Thed N. van Leeuwen,et al.  Some modifications to the SNIP journal impact indicator , 2012, J. Informetrics.

[42]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[43]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[44]  Kevin W. Boyack,et al.  Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately? , 2010 .

[45]  E Garfield,et al.  Long-term vs. short-term journal impact: does it matter? , 1998, The Physiologist.

[46]  Dejan Pajic,et al.  Globalization of the social sciences in Eastern Europe: genuine breakthrough or a slippery slope of the research evaluation practice? , 2015, Scientometrics.

[47]  Emilio Delgado López-Cózar,et al.  Ranking journals: could Google Scholar Metrics be an alternative to Journal Citation Reports and Scimago Journal Rank? , 2013, Learn. Publ..

[48]  Ismael Rafols,et al.  How journal rankings can suppress interdisciplinarity. The case of innovation studies in business and management , 2011, ArXiv.

[49]  Juan Miguel Campanario.  Empirical study of journal impact factors obtained using the classical two-year citation window versus a five-year citation window , 2010, Scientometrics.

[50]  Massimo Franceschet,et al.  Journal influence factors , 2010, J. Informetrics.

[51]  C. Harvey,et al.  Accounting journals and the ABS Guide: a review of evidence and inference , 2013 .

[52]  Jerome K. Vanclay,et al.  An evaluation of the Australian Research Council's journal ranking , 2010, J. Informetrics.

[53]  Vicente P. Guerrero-Bote,et al.  A further step forward in measuring journals' scientific prestige: The SJR2 indicator , 2012, J. Informetrics.

[54]  Eric H. J. Spruyt,et al.  Changing publication patterns in the Social Sciences and Humanities, 2000–2009 , 2012, Scientometrics.

[55]  Dejan Pajić.  Journal classifications based on citation data: The comparison and suitability of three distance measures , 2013 .