Normalisation of citation impact in economics

This study is intended to facilitate fair research evaluations in economics. Field- and time-normalisation of citation impact is the standard method in bibliometrics. Since citation rates for journal papers differ substantially across publication years and Journal of Economic Literature classification codes, citation rates should be normalised for the comparison of papers across different time periods and economic subfields. Without normalisation, both factors that are independent of research quality might lead to misleading results of citation analyses. We apply two normalised indicators in economics, which are the most important indicators in bibliometrics: (1) the mean normalised citation score (MNCS) compares the citation impact of a focal paper with the mean impact of similar papers published in the same economic subfield and publication year. (2) PPtop 10 % is the share of papers that belong to the 10% most cited papers in a certain subfield and time period. Since the MNCS is based on arithmetic averages despite skewed citation distributions, we recommend using PPtop 10 % for fair comparisons of entities in economics. In this study, we apply the normalisation methods to 294 journals (including normalised scores for 192,524 papers). We used the PPtop 10 % results for assigning the journals to four citation impact classes. Seventeen journals have been identified as outstandingly cited. Two journals, Quarterly Journal of Economics and Journal of Economic Literature, perform statistically significantly better than all other journals. Thus, only two journals can be clearly separated from the rest in economics.

[1]  Javier Ruiz-Castillo Ucelay.  The evaluation of citation distributions , 2012 .

[2]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[3]  S. Liebowitz,et al.  Assessing the Relative Impacts of Economic Journals , 1984 .

[4]  Henk F. Moed,et al.  Comprehensive indicator comparisons intelligible to non-experts: the case of two SNIP versions , 2015, Scientometrics.

[5]  Christian Zimmermann,et al.  Academic Rankings with RePEc , 2012 .

[6]  Henk F. Moed,et al.  Multidimensional assessment of scholarly research impact , 2014, J. Assoc. Inf. Sci. Technol..

[7]  A. Coats.  The Role of Scholarly Journals in the History of Economics: An Essay , 1971 .

[8]  Benno Torgler,et al.  Citation success over time: theory or empirics? , 2012, Scientometrics.

[9]  Tibor Braun,et al.  Relative indicators and relational charts for comparative assessment of publication output and citation impact , 1986, Scientometrics.

[10]  Stephen J. Bensman.  The evaluation of research by scientometric indicators , 2011, J. Assoc. Inf. Sci. Technol..

[11]  Lutz Bornmann,et al.  Sampling issues in bibliometric analysis: Response to discussants , 2016, J. Informetrics.

[12]  Beatrice Cherrier,et al.  Classifying Economics: A History of the JEL Codes , 2015 .

[13]  Kevin W Boyack,et al.  Mapping knowledge domains: Characterizing PNAS , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[15]  John M. Gowdy,et al.  Evolution as a general theoretical framework for economics and public policy , 2013 .

[16]  Glenn Ellison,et al.  Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship , 2017 .

[17]  Andreas Strotmann,et al.  Combining commercial citation indexes and open-access bibliographic databases to delimit highly interdisciplinary research fields for citation analysis , 2010, J. Informetrics.

[18]  Lutz Bornmann,et al.  The wisdom of citing scientists , 2013, J. Assoc. Inf. Sci. Technol..

[19]  Igor Podlubny,et al.  A note on comparison of scientific impact expressed by number of citations in different fields of science , 2004, ArXiv.

[20]  Claudio Castellano,et al.  Rescaling citations of publications in Physics , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  Lutz Bornmann,et al.  On the causes of subject-specific citation rates in Web of Science , 2014, Scientometrics.

[22]  David I. Stern,et al.  Uncertainty Measures for Economics Journal Impact Factors , 2012 .

[23]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[24]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[25]  Javier Ruiz-Castillo,et al.  The evaluation of citation distributions , 2011 .

[26]  Ludo Waltman.  Special section on statistical inference in citation analysis , 2016, J. Informetrics.

[27]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[28]  Loet Leydesdorff,et al.  A review of theory and practice in scientometrics , 2015, Eur. J. Oper. Res..

[29]  Lutz Bornmann,et al.  Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes , 2011, J. Assoc. Inf. Sci. Technol..

[30]  Tobias Opthof,et al.  Differences in citation frequency of clinical and basic science papers in cardiovascular research , 2011, Medical & Biological Engineering & Computing.

[31]  Lutz Bornmann,et al.  An empirical look at the nature index , 2017, J. Assoc. Inf. Sci. Technol..

[32]  Mike Thelwall,et al.  Three practical field normalised alternative indicator formulae for research evaluation , 2016, J. Informetrics.

[33]  Jesper W. Schneider,et al.  The imaginarium of statistical inference when data are the population: Comments to Williams and Bornmann , 2016, J. Informetrics.

[34]  John Gibson,et al.  Citations or Journal Quality: Which is Rewarded More in the Academic Labor Market? , 2017 .

[35]  Ludo Waltman,et al.  A systematic empirical comparison of different approaches for normalizing citation impact indicators , 2013, J. Informetrics.

[36]  Laurent Linnemer,et al.  Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics , 2010 .

[37]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[38]  W. Glänzel.  Seven Myths in Bibliometrics About facts and fiction in quantitative science studies , 2008 .

[39]  Thed N. van Leeuwen,et al.  Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university , 2015, Scientometrics.

[40]  Andrea Bonaccorsi,et al.  How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise , 2016 .

[41]  James A. Robinson,et al.  The Colonial Origins of Comparative Development: An Empirical Investigation , 2000 .

[42]  Ludo Waltman,et al.  Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison , 2012, Scientometrics.

[43]  Ludo Waltman,et al.  On the calculation of percentile-based bibliometric indicators , 2012, J. Assoc. Inf. Sci. Technol..

[44]  Loet Leydesdorff,et al.  Citation Analysis with Medical Subject Headings (MeSH) using the Web of Knowledge: A new routine , 2012, J. Assoc. Inf. Sci. Technol..

[45]  James G. Corrigan,et al.  Programmatic evaluation and comparison based on standardized citation scores , 1983, IEEE Transactions on Engineering Management.

[46]  Peter Vinkler,et al.  Evaluation of some methods for the relative assessment of scientific publications , 1986, Scientometrics.

[47]  Thed N. van Leeuwen,et al.  Redefining the field of economics: Improving field normalization for the application of bibliometric techniques in the field of economics , 2012 .

[48]  Jonas Lundberg,et al.  Lifting the crown - citation z-score , 2007, J. Informetrics.

[49]  Loet Leydesdorff,et al.  Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data , 2016, J. Informetrics.

[50]  François Claveau.  There should not be any mystery: A comment on sampling issues in bibliometrics , 2016, J. Informetrics.

[51]  Qi Wang,et al.  Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus , 2015, J. Informetrics.

[52]  Yunrong Li,et al.  The impact of extreme observations in citation distributions , 2014 .

[53]  Wolfgang Glänzel,et al.  Characteristic scores and scales in assessing citation impact , 1988, J. Inf. Sci..

[54]  Rüdiger Mutz,et al.  Some further aspects of sampling: Comment on Williams and Bornmann , 2016, J. Informetrics.

[55]  K. Gwet.  Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters , 2014 .

[56]  Wolfgang Glänzel,et al.  Applying the CSS method to bibliometric indicators used in (university) rankings , 2016, Scientometrics.

[57]  Lutz Bornmann,et al.  How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations , 2013, Scientometrics.

[58]  Anthony F. J. van Raan,et al.  Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure , 2005 .

[59]  Douglas G. Altman,et al.  Statistics with confidence: Confidence intervals and statistical guidelines . , 1990 .

[60]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[61]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[62]  Lea-Rachel Kosnik,et al.  A Survey of JEL Codes: What Do They Mean and are They Used Consistently? , 2018 .

[63]  Tony Lawson,et al.  What is this ‘school’ called neoclassical economics? , 2013 .

[64]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[65]  Javier Ruiz-Castillo,et al.  The effect on citation inequality of differences in citation practices at the web of science subject category level , 2014, J. Assoc. Inf. Sci. Technol..

[66]  Pierre Azoulay,et al.  Economic Research Evolves: Fields and Styles , 2017 .

[67]  Oscar Volij,et al.  The Measurement of Intellectual Influence , 2002 .

[68]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[69]  Lutz Bornmann,et al.  What are the top five journals in economics? A new meta-ranking , 2017 .

[70]  Igor Podlubny,et al.  Comparison of scientific impact expressed by the number of citations in different fields of science , 2004, Scientometrics.

[71]  Gabriel Kreiman,et al.  Nine Criteria for a Measure of Scientific Output , 2011, Front. Comput. Neurosci..

[72]  Philip J. Reny,et al.  How To Count Citations If You Must , 2016 .

[73]  Tindaro Cicero,et al.  Assessing the varying level of impact measurement accuracy as a function of the citation window length , 2011, J. Informetrics.

[74]  E. Garfield.  The history and meaning of the journal impact factor. , 2006, JAMA.

[75]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[76]  Hairul Azlan Annuar,et al.  Foreign investors' interests and corporate tax avoidance: Evidence from an emerging economy , 2015 .

[77]  Gaby Haddow,et al.  Misfits? Research classification in research evaluation: visualizing journal content within fields of research codes , 2013 .

[78]  D. Hamermesh.  Citations in Economics: Measurement, Uses and Impacts , 2015, SSRN Electronic Journal.

[79]  B. Martin,et al.  Assessing Basic Research : Some Partial Indicators of Scientific Progress in Radio Astronomy : Research Policy , 1987 .

[80]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[81]  Sebastian Galiani,et al.  Quantifying the Life Cycle of Scholarly Articles Across Fields of Economic Research , 2016 .

[82]  Lawrence J. Smolinsky,et al.  Citation rates in mathematics: a study of variation by subdiscipline , 2012, Scientometrics.

[83]  Lutz Bornmann,et al.  Sampling issues in bibliometric analysis , 2014, J. Informetrics.

[84]  John Gibson,et al.  Which Journal Rankings Best Explain Academic Salaries? Evidence from the University of California , 2012 .

[85]  Koenraad Debackere,et al.  Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance , 2008, Scientometrics.

[86]  Michel Gevers.  Scientific performance indicators: a critical appraisal and a country- by-country analysis , 2014 .

[87]  Ronald N. Kostoff,et al.  Citation analysis of research performer quality , 2004, Scientometrics.

[88]  Lutz Bornmann,et al.  Selecting manuscripts for a high-impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere , 2008, J. Assoc. Inf. Sci. Technol..

[89]  Geoff Cumming,et al.  Introduction to the New Statistics: Estimation, Open Science, and Beyond , 2016 .

[90]  Glenn Ellison,et al.  How Does the Market Use Citation Data? The Hirsch Index in Economics , 2010, SSRN Electronic Journal.

[91]  Lutz Bornmann,et al.  What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide , 2014, J. Informetrics.