On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff

Opthof and Leydesdorff (Scientometrics, 2011) reanalyze data reported by Van Raan (Scientometrics 67(3):491–502, 2006) and conclude that there is no significant correlation between on the one hand average citation scores measured using the CPP/FCSm indicator and on the other hand the quality judgment of peers. We point out that Opthof and Leydesdorff draw their conclusions based on a very limited amount of data. We also criticize the statistical methodology used by Opthof and Leydesdorff. Using a larger amount of data and a more appropriate statistical methodology, we do find a significant correlation between the CPP/FCSm indicator and peer judgment.

[1]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[2]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[3]  Loet Leydesdorff,et al.  Citation analysis cannot legitimate the strategic selection of excellence , 2011, ArXiv.

[4]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[5]  Loet Leydesdorff,et al.  A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011 , 2011, Scientometrics.

[6]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[7]  L. Adler.  A Modification of Kendall's Tau for the Case of Arbitrary Ties in Both Rankings , 1957 .

[8]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[9]  Rex B. Kline,et al.  Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research , 2004 .

[10]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[11]  Anthony F. J. van Raan.  Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups , 2013, Scientometrics.

[12]  M. E. Bustamante,et al.  A comment on this paper. , 1965 .

[13]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[14]  A modification of kendall's tau for measuring association in contingency tables , 1958 .