Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?

This study uses journal ratings in criminology and criminal justice, library and information science, public administration, and social work to investigate two research questions: (1) Are stated preference (subjective) journal ratings more closely related to size-dependent citation metrics (eigenfactor and total citations, which represent the impact of the journal as a whole) or to size-independent citation metrics (article influence and CiteScore, which represent the impact of a typical article)? (2) Are stated preference ratings more closely related to unweighted citation metrics (five-year impact factor and source normalized impact per publication, which do not account for the impact of each citing journal) or to weighted citation metrics (article influence and SCImago journal rank, which do)? Within the disciplines evaluated here, respondents’ subjective ratings of journals are more closely related to size-independent metrics and weighted metrics. The relative strength of the relationship between subjective ratings and size-independent metrics is moderated by subject area and other factors, while the relative strength of the relationship between subjective ratings and weighted metrics is consistent across all four disciplines. These results are discussed with regard to popularity and prestige, which are sometimes associated with unweighted and weighted citation metrics, respectively.

[1]  Johan Bollen,et al.  Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics , 2006, J. Assoc. Inf. Sci. Technol..

[2]  Robert H. Luke,et al.  Marketing journal hierarchies: Faculty perceptions, 1986-87 , 1987 .

[3]  Vicente P. Guerrero-Bote,et al.  A further step forward in measuring journals' scientific prestige: The SJR2 indicator , 2012, J. Informetrics.

[4]  John Mingers,et al.  Ranking journals in business and management: a statistical analysis of the Harzing data set , 2007, Eur. J. Inf. Syst..

[5]  Laura Manzari,et al.  Library and Information Science Journal Prestige as Assessed by Library and Information Science Faculty , 2013, The Library Quarterly.

[6]  Pairin Katerattanakul,et al.  CONSISTENCY AND CONCERN ON IS JOURNAL RANKINGS , 2005 .

[7]  Michael F. Gorman,et al.  Evaluating Operations Management-Related Journals via the Author Affiliation Index , 2005, Manuf. Serv. Oper. Manag..

[8]  William H. Walters,et al.  Beyond use statistics: Recall, precision, and relevance in the assessment and management of academic libraries , 2016, J. Libr. Inf. Sci..

[9]  H. Goldstein,et al.  The Use and Valuation of Journals in Planning Scholarship: Peer Assessment versus Impact Factors , 2010 .

[10]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[11]  R. Rentschler,et al.  Assessing Sport Management Journals: A Multi-Dimensional Examination , 2007 .

[12]  Péter Jacsó,et al.  The h‐index, h‐core citation rate and the bibliometric profile of the Scopus database , 2011 .

[13]  Waldemar Karwowski,et al.  Objective and Subjective Rankings of Scientific Journals in the Field of Ergonomics: 2004-2005 , 2005 .

[14]  Loet Leydesdorff.  How are new citation-based journal indicators adding to the bibliometric toolbox? , 2009 .

[15]  Michael C. Calver,et al.  Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field? , 2009, Scientometrics.

[16]  Wolfgang G. Stock,et al.  Impact and relevance of LIS journals: A scientometric analysis of international and German-language LIS journals - Citation analysis versus reader survey , 2004, J. Assoc. Inf. Sci. Technol..

[17]  Henk F. Moed,et al.  Assessing the quality of scholarly journals in Linguistics:An alternative to citation-based journal impact factors , 2001, Scientometrics.

[18]  Cassidy R. Sugimoto,et al.  P-Rank: An indicator measuring prestige in heterogeneous scholarly networks , 2011, J. Assoc. Inf. Sci. Technol..

[19]  Thomas R. Miller,et al.  Ratings of Professional Journals by Teachers of Management. , 1979 .

[20]  Ying Ding,et al.  Weighted citation: An indicator of an article's prestige , 2010, J. Assoc. Inf. Sci. Technol..

[21]  Ted Bergstrom Papers.  The Eigenfactor Metrics: A network approach to assessing scholarly journals , 2010 .

[22]  E. Susanna Cahn,et al.  Journal Rankings: Comparing Reputation, Citation and Acceptance Rates , 2014, Int. J. Inf. Syst. Serv. Sect..

[23]  James C. Garand,et al.  Ranking Political Science Journals: Reputational and Citational Approaches , 2007, PS: Political Science & Politics.

[24]  C. Harvey,et al.  Journal rankings and the ABS Journal Quality Guide , 2009 .

[25]  Ben R. Martin,et al.  The use of multiple indicators in the assessment of basic research , 1996, Scientometrics.

[26]  Jerome K. Vanclay,et al.  Ranking forestry journals using the h-index , 2007, J. Informetrics.

[27]  Ben Wooliscroft,et al.  Measuring the impact of accounting journals using Google Scholar and the g-index , 2009 .

[28]  Peter Haddawy,et al.  A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality , 2016, J. Informetrics.

[29]  V. Larivière.  The limits of bibliometrics for the analysis of the social sciences and humanities literature , 2010 .

[30]  Massimo Franceschet,et al.  A cluster analysis of scholar and journal bibliometric indicators , 2009, J. Assoc. Inf. Sci. Technol..

[31]  E. Cohn,et al.  Scholarly Influence and Prestige in Criminology and Criminal Justice , 2011 .

[32]  Cassidy R. Sugimoto,et al.  Visualizing and comparing four facets of scholarly communication: producers, artifacts, concepts, and gatekeepers , 2012, Scientometrics.

[33]  E. E. Nkereuwem.  Accrediting knowledge: The ranking of library and information science journals , 1997 .

[34]  J. Hudson.  Ranking Journals , 2013 .

[35]  Josephine E. Olson,et al.  Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields , 2005, Interfaces.

[36]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[37]  Scott L. B. McConnell,et al.  Research Quality Rankings of Heterodox Economic Journals in a Contested Discipline , 2010 .

[38]  Robin Perry,et al.  Evaluation of Social Work Journal Quality: Citation versus Reputation Approaches , 2004 .

[39]  Massimo Franceschet,et al.  The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis , 2010, J. Informetrics.

[40]  M. Polonsky,et al.  What Are We Measuring When We Evaluate Journals? , 2005 .

[41]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[42]  Thed N. van Leeuwen,et al.  Some modifications to the SNIP journal impact indicator , 2012, J. Informetrics.

[43]  T. E. Nisonger,et al.  The Perception of Library and Information Science Journals by LIS Education Deans and ARL Library Directors: A Replication of the Kohl–Davis Study , 2005 .

[44]  Chih-Fong Tsai.  Citation impact analysis of top ranked computer science journals and their rankings , 2014, J. Informetrics.

[45]  Tamara Krajna,et al.  SCImago Journal & Country Rank , 2008 .

[46]  Robert West,et al.  What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. , 2002, Addiction.

[47]  A. Tahai,et al.  A revealed preference study of management journals' direct influences , 1999 .

[48]  Abdullah Abrizah,et al.  LIS journals scientific impact and subject categorization: a comparison between Web of Science and Scopus , 2012, Scientometrics.

[49]  T. M. Nelson,et al.  Rating of scholarly journals by chairpersons in the social sciences , 1983 .

[50]  Christian Gumpenberger,et al.  Going beyond Citations: SERUM — a new Tool Provided by a Network of Libraries , 2010 .

[51]  William H. Walters,et al.  Information Sources and Indicators for the Assessment of Journal Reputation and Impact , 2016 .

[52]  Xu Liu,et al.  Quality-structure index: A new metric to measure scientific journal influence , 2011, J. Assoc. Inf. Sci. Technol..

[53]  Tommi Kärkkäinen,et al.  Expert-based versus citation-based ranking of scholarly and scientific publication channels , 2016, J. Informetrics.

[54]  Jon R. Sorensen,et al.  An Assessment of Criminal Justice and Criminology Journal Prestige , 2006 .

[55]  Judith M. Nixon,et al.  Core Journals in Library and Information Science: Developing a Methodology for Ranking LIS Journals , 2014, Coll. Res. Libr..

[56]  Paul Genoni,et al.  Citation analysis and peer ranking of Australian social science journals , 2010, Scientometrics.

[57]  William H. Walters,et al.  Evaluating Online Resources for College and University Libraries: Assessing Value and Cost Based on Academic Needs , 2016 .

[58]  C. Y. K. So,et al.  Citation ranking versus expert judgment in evaluating communication scholars: Effects of research specialty size and individual prominence , 1998, Scientometrics.

[59]  Carl T. Bergstrom,et al.  The Eigenfactor MetricsTM: A Network Approach to Assessing Scholarly Journals , 2010, Coll. Res. Libr..

[60]  Alexander Serenko,et al.  Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence , 2011, J. Informetrics.

[61]  D. C. Howell.  Statistical Methods for Psychology , 1987 .

[62]  Johan Bollen,et al.  Journal status , 2006, Scientometrics.

[63]  J. Odell,et al.  The Interdisciplinary Influence of Library and Information Science 1996–2004: A Journal-to-Journal Citation Analysis , 2008 .

[64]  Sandra C. Jones,et al.  The development of a ranking tool for refereed journals in which nursing and midwifery researchers publish their work. , 2010, Nurse education today.

[65]  Ying Ding,et al.  Popular and/or prestigious? Measures of scholarly esteem , 2010, Inf. Process. Manag..

[66]  L. Ellis,et al.  Why economists rank their journals the way they do , 1991 .

[67]  T. Lange.  The Imprecise Science of Evaluating Scholarly Performance , 2006, Evaluation review.

[68]  Gunther Maier,et al.  Impact factors and peer judgment: The case of regional science journals , 2006, Scientometrics.

[69]  Jeremy B. Fox,et al.  A multi-method evaluation of journals in the decision and management sciences by US academics , 2000 .

[70]  Ludo Waltman,et al.  The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments , 2014, J. Informetrics.

[71]  Ludo Waltman,et al.  The inconsistency of the h-index , 2011, J. Assoc. Inf. Sci. Technol..

[72]  James C. Garand.  An Alternative Interpretation of Recent Political Science Journal Evaluations , 1990, PS: Political Science & Politics.

[73]  Massimo Franceschet.  A cluster analysis of scholar and journal bibliometric indicators , 2009 .

[74]  Birk Diedenhofen,et al.  cocor: A Comprehensive Solution for the Statistical Comparison of Correlations , 2015, PloS one.

[75]  Miranda Lee Pao,et al.  A discipline-specific journal selection algorithm , 1986, Inf. Process. Manag..

[76]  Vicente P. Guerrero-Bote,et al.  A new approach to the metric of journals' scientific prestige: The SJR indicator , 2010, J. Informetrics.

[77]  Juan Gorraiz,et al.  Comparison of citation and usage indicators: the case of oncology journals , 2010, Scientometrics.

[78]  Jian Ma,et al.  A group decision support approach to evaluating journals , 2004, Inf. Manag..

[79]  Nick Bontis,et al.  What's familiar is excellent: The impact of exposure effect on perceived journal quality , 2011, J. Informetrics.

[80]  J. Christenson.  ACCREDITING KNOWLEDGE: JOURNAL STATURE AND CITAtiON IMPACT IN SOCIAL SCIENCE1 , 1985 .

[81]  Courtney Shelton Hunt,et al.  Not Quite as Simple as A-B-C , 2006 .

[82]  Johan Bollen,et al.  A Principal Component Analysis of 39 Scientific Impact Measures , 2009, PloS one.

[83]  Jon R. Sorensen.  An assessment of the relative impact of criminal justice and criminology journals , 2009 .

[84]  Thomas E. Nisonger,et al.  The Benefits and Drawbacks of Impact Factor for Journal Collection Management in Libraries , 2004 .

[85]  Sherrill L. Sellers,et al.  PERCEPTIONS OF PROFESSIONAL SOCIAL WORK JOURNALS: FINDINGS FROM A NATIONAL SURVEY , 2006 .

[86]  Loet Leydesdorff,et al.  How are new citation-based journal indicators adding to the bibliometric toolbox? , 2009, J. Assoc. Inf. Sci. Technol..

[87]  E. J. Williams.  The Comparison of Regression Variables , 1959 .