Ranking Law Journals and the Limits of Journal Citation Reports

Rankings of schools, scholars, and journals emphasize ordinal rank. Journal rankings published by Journal Citation Reports (JCR) are widely used to assess research quality, which influences important decisions by academic departments, universities, and countries. We study refereed law journal rankings by JCR, Washington and Lee Law Library (W&L), and the Australian Research Council (ARC). Both JCR’s and W&L’s multiple measures of journals can be represented by a single latent factor. Yet JCR’s rankings are uncorrelated with W&L’s. The differences appear to be attributable to underrepresentation of law journals in JCR’s database. We illustrate the effects of database bias on rankings through case studies of three elite journals, the Journal of Law & Economics, Supreme Court Review, and the American Law & Economics Review. Cluster analysis is a supplement to ordinal ranking and we report the results of a cluster analysis of law journals. The ARC does organize journals into four large groups and provides generally reasonable rankings of journals. But anomalies exist that could be avoided by checking the ARC groups against citation-based measures. Entities that rank should use their data to provide meaningful clusters rather than providing only ordinal ranks.

[1]  Carl T. Bergstrom,et al.  Author-Level Eigenfactor Metrics: Evaluating the Influence of Authors, Institutions and Countries Within the SSRN Community , 2012 .

[2]  Imad A. Moosa,et al.  The Demise of the ARC Journal Ranking Scheme: An Ex Post Analysis of the Accounting and Finance Journals , 2011 .

[3]  Russell R. Currie,et al.  Finance Journal Rankings & Tiers: An Active Scholar Assessment Methodology , 2010 .

[4]  Yong Bao,et al.  General-Interest Versus Specialty Journals: Using Intellectual Influence of Econometrics Research to Rank Economics Journals and Articles , 2010 .

[5]  Carl T. Bergstrom,et al.  Big Macs and Eigenfactor scores: Don't let correlation coefficients fool you , 2009, J. Assoc. Inf. Sci. Technol..

[6]  Philip M. Davis Reward or persuasion? The battle to define the meaning of a citation , 2009, Learn. Publ..

[7]  Werner Marx,et al.  The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the Journal Impact Factor , 2009, J. Assoc. Inf. Sci. Technol..

[8]  Philip M. Davis Eigenfactor : Does the Principle of Repeated Improvement Result in Better Journal Impact Estimates than Raw Citation Counts? , 2008, J. Assoc. Inf. Sci. Technol..

[9]  S. Teles The Rise of the Conservative Legal Movement: The Battle for Control of the Law , 2008 .

[10]  E. Garfield Citation indexes for science. A new dimension in documentation through association of ideas. 1955. , 1955, International journal of epidemiology.

[11]  T. Eisenberg Assessing the Ssrn-Based Law School Rankings , 2005 .

[12]  David E. Booth,et al.  Applied Multivariate Analysis , 2003, Technometrics.

[13]  Oscar Volij,et al.  The Measurement of Intellectual Influence , 2002 .

[14]  Wolfgang Glänzel,et al.  How balanced is the Science Citation Index's journal coverage? A preliminary overview of macrolevel statistical data , 2000 .

[15]  M. Wells,et al.  Ranking and Explaining the Scholarly Impact of Law Schools , 1998, The Journal of Legal Studies.

[16]  P. Sopp Cluster analysis. , 1996, Veterinary immunology and immunopathology.

[17]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[18]  E. Garfield Citation analysis as a tool in journal evaluation. , 1972, Science.

[19]  E. Garfield,et al.  Citation indexes for science. , 1956, Science.