Citation analysis and peer ranking of Australian social science journals

Citation analyses were performed for Australian social science journals to determine the differences between data drawn from Web of Science and Scopus. These data were compared with the tier rankings assigned by disciplinary groups to the journals for the purposes of a new research assessment model, Excellence in Research for Australia (ERA), due to be implemented in 2010. In addition, citation-based indicators including an extended journal impact factor, the h-index, and a modified journal diffusion factor, were calculated to assess whether subsequent analyses influence the ranking of journals. The findings suggest that the Scopus database provides higher number of citations for more of the journals. However, there appears to be very little association between the assigned tier ranking of journals and their rank derived from citations data. The implications for Australian social science researchers are discussed in relation to the use of citation analysis in the ERA.

[1]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[2]  Charles Oppenheim,et al.  Citation counts and the Research Assessment Exercise, part VI: Unit of assessment 67 (music) , 2008, Inf. Res..

[3]  Narongrit Sombatsompop,et al.  Article-count impact factor of materials science journals in SCI database , 2008, Scientometrics.

[4]  Alastair G. Smith,et al.  Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise , 2008, Scientometrics.

[5]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[6]  Kerry Smith,et al.  Australian Library & Information Studies (LIS) Researchers Ranking of LIS Journals. , 2009 .

[7]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[8]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems , 2004 .

[9]  Pam Royle,et al.  The Use of Bibliometric Indicators to Measure the Research Productivity of Australian Academics , 1994 .

[10]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..

[11]  Julian Warner,et al.  A critical review of the application of citation studies to the Research Assessment Exercises , 2000, J. Inf. Sci..

[12]  Linda Butler,et al.  Extending citation analysis to non-source items , 2006, Scientometrics.

[13]  Carnot E. Nelson,et al.  Communication among scientists and engineers , 1970 .

[14]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[15]  H. Browman,et al.  Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely , 2008 .

[16]  Paul Genoni,et al.  Australian Education Journals: Quantitative and Qualitative Indicators , 2009 .

[17]  Lei Wang,et al.  Three options for citation tracking: Google Scholar, Scopus and Web of Science , 2006, Biomedical digital libraries.

[18]  Diana Hicks,et al.  The difficulty of achieving full coverage of international social science literature and the bibliometric consequences , 1999, Scientometrics.

[19]  P. Jacsó.  As we may search : Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases , 2005 .

[20]  Tove Faber Frandsen,et al.  Journal diffusion factors - a measure of diffusion? , 2004, Aslib Proc..

[21]  John W. East,et al.  Ranking Journals in the Humanities: An Australian Case Study , 2006 .

[22]  Maurice B. Line,et al.  The structure of social science literature as shown by a large-scale citation analysis , 1981 .

[23]  D. Hicks.  The Four Literatures of Social Science , 2004 .

[24]  Ian Rowlands,et al.  Journal diffusion factors: a new approach to measuring research influence , 2002, Aslib Proc..

[25]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007 .

[26]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[27]  Brian D. Cameron,et al.  Trends in the Usage of ISI Bibliometric Data: Uses, Abuses, and Implications , 2005 .

[28]  Queensland,et al.  Australian and New Zealand Standard Research Classification , 2006 .

[29]  Linda Butler,et al.  The publishing imperative: the pervasive influence of publication metrics , 2006, Learn. Publ..

[30]  B. Vickery,et al.  SOCIAL SCIENCE LITERATURE USE IN THE UK AS INDICATED BY CITATIONS , 1969 .

[31]  Russell Smyth,et al.  A Citation Analysis of Australian Economic Journals , 1999 .

[32]  Pam Royle.  A Citation Analysis of Australian Science and Social Science Journals , 1994 .

[33]  R. M. Hayes.  Citation Statistics as a Measure of Faculty Research Productivity , 1983 .

[34]  Lars Iselid,et al.  Web of Science and Scopus: a journal title overlap study , 2008, Online Inf. Rev..

[35]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[36]  Charles Oppenheim,et al.  The Correlation between citation counts and the 1992 Research Assessment Exercise Ratings for British Library and Information Science University departments , 1995, J. Documentation.

[37]  Charles Oppenheim,et al.  Use of citation analysis to predict the outcome of the 2001 Research Assessment Exercise for Unit of Assessment (UoA) 61: Library and Information Management, , 2001, Inf. Res..

[38]  Charles Oppenheim,et al.  The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology , 1997, J. Documentation.

[39]  Linda Butler,et al.  Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework , 2008 .

[40]  Paul Genoni,et al.  ERA and the ranking of Australian humanities journals , 2009 .

[41]  Nisa Bakkalbasi,et al.  An Examination of Citation Counts in a New Scholarly Communication Environment , 2005, D Lib Mag..

[42]  Olle Persson,et al.  The DCI index: Discounted cumulated impact-based research evaluation , 2008 .

[43]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Google Scholar , 2008, Online Inf. Rev..

[44]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[45]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[46]  Jeroen Bosman,et al.  Scopus reviewed and compared: the coverage and functionality of the citation database Scopus, including comparisons with Web of Science and Google Scholar , 2006 .

[47]  Gaby Haddow.  Quality Australian Journals in the Humanities and Social Sciences , 2008 .

[48]  Charles Oppenheim,et al.  Comparing alternatives to the Web of Science for coverage of the social sciences' literature , 2007, J. Informetrics.