Research assessment using early citation information

Peer-review based research assessment, as implemented in Australia, the United Kingdom, and some other countries, is a very costly exercise. We show that university rankings in economics based on long-run citation counts can be easily predicted using early citations. This would allow a research assessment to predict the relative long-run impact of articles published by a university immediately at the end of the evaluation period. We compare these citation-based university rankings with the rankings of the 2010 Excellence in Research assessment in Australia and the 2008 Research Assessment Exercise in the United Kingdom. Rank correlations are quite strong, but there are some differences between rankings. However, if assessors are willing to consider citation analysis to assess some disciplines, as is the case for the natural sciences and psychology in Australia, it seems reasonable to consider also including economics in that set.

[1]  T. Jappelli,et al.  Bibliometric Evaluation vs. Informed Peer Review: Evidence from Italy , 2013, SSRN Electronic Journal.

[2]  F. Neri,et al.  The Contribution of Australian Academia to the World's Best Economics Research: 2001 to 2010 , 2015 .

[3]  M. Pesaran,et al.  Testing for unit roots in heterogeneous panels , 2003 .

[4]  Ralph Kenna,et al.  Critical mass and the dependency of research quality on group size , 2010, Scientometrics.

[5]  Charles Oppenheim,et al.  Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE , 2003, J. Documentation.

[6]  Lutz Bornmann,et al.  How much does the expected number of citations for a publication change if it contains the address of a specific scientific institute? A new approach for the analysis of citation data on the institutional level based on regression models , 2016, J. Assoc. Inf. Sci. Technol..

[7]  Jonathan Adams,et al.  Early citation counts correlate with accumulated impact , 2005, Scientometrics.

[8]  Olesya Mryglod,et al.  Predicting results of the Research Excellence Framework using departmental h-index , 2014, Scientometrics.

[9]  Andrew M. Colman,et al.  A bibliometric evaluation of the research performance of British university politics departments: Publications in leading journals , 2005, Scientometrics.

[10]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[11]  Jeremiah P. Ostriker,et al.  Committee on an Assessment of Research-Doctorate Programs , 2011 .

[12]  Andrew J. Oswald,et al.  How Should Peer�?Review Panels Behave? , 2013 .

[13]  John D. Tressler,et al.  The New Zealand performance-based research fund and its impact on publication activity in economics , 2013 .

[14]  Alan Reeves,et al.  Has economics become an elite subject for elite UK universities? , 2014 .

[15]  Steven Wooding,et al.  UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim? , 2015, PloS one.

[16]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[17]  J. Hudson.  Ranking Journals , 2013 .

[18]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[19]  Albert-László Barabási,et al.  Quantifying Long-Term Scientific Impact , 2013, Science.

[20]  Katharine Rockett,et al.  A Tale of Two Metrics: Research Assessment vs Recognised Excellence , 2014 .

[21]  Martin Steininger,et al.  Towards a European economics of economics: Monitoring a decade of top research and providing some explanation , 2006, Scientometrics.

[22]  Jeremiah P. Ostriker,et al.  A Data-Based Assessment of Research-Doctorate Programs in the United States , 2011 .

[23]  Mike Thelwall,et al.  The metric tide: report of the independent review of the role of metrics in research assessment and management , 2015 .

[24]  Olesya Mryglod,et al.  Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence , 2013, Scientometrics.

[25]  David I. Stern,et al.  High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information , 2014, PloS one.

[26]  Loet Leydesdorff,et al.  Scientometrics in a changing research landscape , 2014, EMBO reports.

[27]  David Stuart,et al.  Rank Hypocrisies: The Insult of the REF , 2016, Online Inf. Rev..

[28]  Charles Oppenheim.  Do Citations Count? Citation Indexing and the Research Assessment Exercise (RAE) , 1996 .

[29]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[30]  Stephen T Holgate.  A comment on “Scientometrics in a changing research landscape” , 2015, EMBO reports.

[31]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[32]  Stephen A. Gallo,et al.  The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies , 2014, PloS one.

[33]  Ludo Waltman,et al.  Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison , 2012, Scientometrics.

[34]  James Wilsdon.  The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management , 2016 .

[35]  Mike Thelwall,et al.  A combined bibliometric indicator to predict article impact , 2011, Inf. Process. Manag..

[36]  Paul Simmonds,et al.  REF Accountability Review: Costs, benefits and burden , 2015 .

[37]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[38]  Alexandros Polycarpou,et al.  Peer Review vs Metric‐Based Assessment: Testing for Bias in the Rae Ratings of UK Economics Departments , 2009 .