Peer Review versus Citations - An Analysis of Best Paper Prizes

In this paper, I analyze the ‘best paper’ prizes given by economics and finance journals to the best article published in their journal in a given year. More specifically, I compare the citations received by best paper prize-winning papers to citations received by papers that are awarded runner up prizes and to citations received by non-winning papers. In this way, I evaluate to what extent evaluation outcomes based on peer review correspond to evaluation outcomes based on citation counts. The data show that the paper that gets the ‘best paper’ prize, is rarely the most cited paper; is, in a small majority of cases, cited more than the runner up papers and is, in most cases, cited more than the median paper. I also explore whether characteristics of the prizes or the papers correlate with this difference in outcomes between peer review and citation counts and find there is no easy way to reduce the difference in outcomes between these two evaluation methods

[1]  Ivy E. Broder,et al.  PROFESSIONAL ACHIEVEMENTS AND GENDER DIFFERENCES AMONG ACADEMIC ECONOMISTS , 1993 .

[2]  A. Neely,et al.  Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001 , 2008 .

[3]  Jerry A. Hausman,et al.  Assessing the potential demand for electric cars , 1981 .

[4]  Peter Schmidt,et al.  The determinants of econometric society fellows elections , 2003 .

[5]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[6]  Laurent Linnemer,et al.  Publish or peer-rich ? The role of skills and networks in hiring economics professors ☆ , 2008 .

[7]  J. Gans,et al.  How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists , 1994 .

[8]  R. Blank.  The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[9]  Laurent Linnemer,et al.  Publish or Peer-rich , 2008 .

[10]  D. Laband.  Is There Value-Added from the Review Process in Economics?: Preliminary Evidence from Authors , 1990 .

[11]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[12]  Eric P. Chiang,et al.  Citation Counts and SSCI in Personnel Decisions: A Survey of Economics Departments , 2004 .

[13]  T. Coupé,et al.  Does the Academic Labor Market Initially Allocate New Graduates Efficiently , 2006 .

[14]  Robert D. Tollison,et al.  Dry Holes in Economic Research , 2003 .

[15]  Koenraad Debackere,et al.  Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key , 2004, Scientometrics.

[16]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[17]  Lutz Bornmann,et al.  Does the Journal Peer Review Select the “Best” from the Work Submitted? The State of Empirical Research , 2010 .

[18]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.