A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011

In reaction to a previous critique (Opthof and Leydesdorff, J Informetr 4(3):423–430, 2010), the Center for Science and Technology Studies (CWTS) in Leiden proposed to change their old “crown” indicator in citation analysis into a new one. Waltman (Scientometrics 87:467–481, 2011a) argue that this change does not affect rankings at various aggregated levels. However, CWTS data is not publicly available for testing and criticism. Therefore, we comment by using previously published data of Van Raan (Scientometrics 67(3):491–502, 2006) to address the pivotal issue of how the results of citation analysis correlate with the results of peer review. A quality parameter based on peer review was neither significantly correlated with the two parameters developed by the CWTS in the past citations per paper/mean journal citation score (CPP/JCSm) or CPP/FCSm (citations per paper/mean field citation score) nor with the more recently proposed h-index (Hirsch, Proc Natl Acad Sci USA 102(46):16569–16572, 2005). Given the high correlations between the old and new “crown” indicators, one can expect that the lack of correlation with the peer-review based quality indicator applies equally to the newly developed ones.

[1]  Loet Leydesdorff,et al.  Caveats for the Use of Citation Indicators in Research and Journal Evaluations , 2008, J. Assoc. Inf. Sci. Technol..

[2]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1992, Scientometrics.

[3]  Michael H. MacRoberts,et al.  Problems of citation analysis: A study of uncited and seldom-cited influences , 2010, J. Assoc. Inf. Sci. Technol..

[4]  Loet Leydesdorff,et al.  Past performance, peer review and project selection: a case study in the social and behavioral sciences , 2009, 0911.1306.

[5]  Willi Meier,et al.  Chemistry and Chemical Engineering , 1999 .

[6]  Peter Vinkler,et al.  Evaluation of some methods for the relative assessment of scientific publications , 1986, Scientometrics.

[7]  Loet Leydesdorff,et al.  Normalization at the field level: fractional counting of citations , 2010, J. Informetrics.

[8]  Loet Leydesdorff,et al.  Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities , 2010, Minerva.

[9]  Peter Vinkler,et al.  Model for quantitative selection of relative scientometric impact indicators , 1996, Scientometrics.

[10]  L. Leydesdorff.  Caveats for the use of citation indicators in research and journal evaluations , 2008 .

[11]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[12]  Loet Leydesdorff,et al.  Remaining problems with the "New Crown Indicator" (MNCS) of the CWTS , 2010, J. Informetrics.

[13]  Tibor Braun,et al.  Relative indicators and relational charts for comparative assessment of publication output and citation impact , 1986, Scientometrics.

[14]  Loet Leydesdorff,et al.  Remaining problems with the , 2010 .

[15]  B. Martin,et al.  University Research Evaluation and Funding: An International Comparison , 2003 .

[16]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[17]  M. H. MacRoberts,et al.  Problems of citation analysis: A study of uncited and seldom-cited influences , 2010 .

[18]  Loet Leydesdorff,et al.  Article in Press G Model Journal of Informetrics a Meta-evaluation of Scientific Research Proposals: Different Ways of Comparing Rejected to Awarded Applications , 2022 .

[19]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[20]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[21]  Jörg Neufeld,et al.  Informed peer review and uninformed bibliometrics , 2011 .

[22]  Thed N. van Leeuwen,et al.  New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications , 1995, Scientometrics.

[23]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[24]  Michael H. MacRoberts,et al.  Another test of the normative theory of citing , 1987, J. Am. Soc. Inf. Sci..

[25]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[26]  Anthony F. J. van Raan.  Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups , 2013, Scientometrics.