Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence

The purpose of this study is to: (1) develop a ranking of peer-reviewed AI journals; (2) compare the consistency of journal rankings developed with two dominant ranking techniques, expert surveys and journal impact measures; and (3) investigate the consistency of journal ranking scores assigned by different categories of expert judges. The ranking was constructed based on the survey of 873 active AI researchers who ranked the overall quality of 182 peer-reviewed AI journals. It is concluded that expert surveys and citation impact journal ranking methods cannot be used as substitutes. Instead, they should be used as complementary approaches. The key problem of the expert survey ranking technique is that in their ranking decisions, respondents are strongly influenced by their current research interests. As a result, their scores merely reflect their present research preferences rather than an objective assessment of each journal's quality. In addition, the application of the expert survey method favors journals that publish more articles per year.

[1]  David S. Watkins,et al.  Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications , 1998, Scientometrics.

[2]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[3]  Robert D. Herbert,et al.  Correlation between the Journal Impact Factor and three other journal citation indices , 2010, Scientometrics.

[4]  A. Sevinç,et al.  Manipulating impact factor: an unethical issue or an Editor's choice? , 2004, Swiss medical weekly.

[5]  William H. Starbuck,et al.  How Much Better are the Most Prestigious Journals? The Statistics of Academic Publication , 2005, Organ. Sci..

[6]  Nick Bontis,et al.  What's familiar is excellent: The impact of exposure effect on perceived journal quality , 2011, J. Informetrics.

[7]  N. Bontis,et al.  A citation-based ranking of the business ethics scholarly journals , 2009 .

[8]  Robin Perry,et al.  Evaluation of Social Work Journal Quality: Citation versus Reputation Approaches , 2004 .

[9]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[10]  Judit Bar-Ilan,et al.  The sources used by bibliometrics-scientometrics as reflected in references , 2002, Scientometrics.

[11]  Henk F. Moed,et al.  Assessing the quality of scholarly journals in Linguistics:An alternative to citation-based journal impact factors , 2001, Scientometrics.

[12]  Richard C. Anderson,et al.  Comparison of peer and citation assessment of the influence of scientific journals , 1980, J. Am. Soc. Inf. Sci..

[13]  Gunther Maier,et al.  Impact factors and peer judgment: The case of regional science journals , 2006, Scientometrics.

[14]  Yen-Liang Chen,et al.  Ranking Taiwanese management journals: A case study , 2008, Scientometrics.

[15]  Ilian A. Bonev.  Should we take journal impact factors seriously , 2009 .

[16]  T. Jacques,et al.  The impact of article titles on citation hits: an analysis of general and specialist medical journals , 2010, JRSM short reports.

[17]  Jane L. Lehr,et al.  Athena Unbound: The Advancement of Women in Science and Technology , 2001 .

[18]  Peter Tarasewich,et al.  Global perceptions of journals publishing e-commerce research , 2002, CACM.

[19]  Linda Butler.  Identifying "highly-rated" journals - an Australian case study , 2004, Scientometrics.

[20]  Nick Bontis,et al.  Global ranking of knowledge management and intellectual capital academic journals , 2009, J. Knowl. Manag..

[21]  Rajiv Kohli,et al.  A multiple criteria assessment of decision technology system journal quality , 2001, Inf. Manag..

[22]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[23]  P. Rogers,et al.  The Impact of Perceptions of Journal Quality on Business and Management Communication Academics , 2007 .

[24]  Sean L. Humpherys,et al.  A Scientometric Study of the Perceived Quality of Business and Technical Communication Journals , 2007, IEEE Transactions on Professional Communication.

[25]  A. Tahai,et al.  A revealed preference study of management journals' direct influences , 1999 .

[26]  M. F. Fox.  Gender, Family Characteristics, and Publication Productivity among Scientists , 2005 .

[27]  Paul Benjamin Lowry,et al.  Global Journal Prestige and Supporting Disciplines: A Scientometric Study of Information Systems Journals , 2004, J. Assoc. Inf. Syst..

[28]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[29]  David B. Balkin,et al.  Determinants of faculty pay: An agency theory perspective. , 1992 .

[30]  Yannis Manolopoulos,et al.  Generalized h-index for Disclosing Latent Facts in Citation Networks , 2006, ArXiv.

[31]  Yannis Manolopoulos,et al.  Generalized Hirsch h-index for disclosing latent facts in citation networks , 2007, Scientometrics.

[32]  Linda M. Manning,et al.  Research Performance Management in Academe , 2009 .

[33]  Jim Suchan.  How Academic Organizational Systems and Culture Undermine Scholarship and Quality Research , 2008 .

[34]  K. Peffers,et al.  Identifying and Evaluating the Universe of Outlets for Information Systems Research: Ranking the Journals , 2003 .

[35]  Jerome K. Vanclay,et al.  Ranking forestry journals using the h-index , 2007, J. Informetrics.

[36]  Ee Nkereuwem,et al.  Citation Ranking Versus Subjective Evaluation In The Determination Of Journal Hierarchies In Information Science , 2008 .

[37]  Paul L. Drnevich,et al.  Does Business School Research Add Economic Value for Students , 2008 .

[38]  Duane P. Truex,et al.  Assessing Scholarly Influence: Using the Hirsch Indices to Reframe the Discourse , 2009, J. Assoc. Inf. Syst..

[39]  John McCarthy,et al.  The Inversion of Functions Defined by Turing Machines , 1956 .

[40]  H. Goldstein,et al.  The Use and Valuation of Journals in Planning Scholarship: Peer Assessment versus Impact Factors , 2010 .

[41]  Bill C. Hardgrave,et al.  Forums for information systems scholars: III , 2001, Inf. Manag..

[42]  Nikolaos A. Mylonopoulos,et al.  On site: global perceptions of IS journals , 2001, CACM.

[43]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..

[44]  Michael C. Calver,et al.  Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field? , 2009, Scientometrics.

[45]  Wolfgang G. Stock,et al.  Impact and relevance of LIS journals: A scientometric analysis of international and German-language LIS journals - Citation analysis versus reader survey , 2004, J. Assoc. Inf. Sci. Technol..

[46]  Clyde W. Holsapple,et al.  Citation-Based Journal Rankings for AI Research A Business Perspective , 1996, AI Mag..

[47]  Waldemar Karwowski,et al.  Objective and Subjective Rankings of Scientific Journals in the Field of Ergonomics: 2004-2005 , 2005 .

[48]  J. C. Korevaar,et al.  Validation of bibliometric indicators in the field of mathematics , 1996, Scientometrics.

[49]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[50]  Massimo Franceschet,et al.  Journal influence factors , 2010, J. Informetrics.

[51]  Clyde W. Holsapple,et al.  An empirical assessment and categorization of journals relevant to DSS research , 1995, Decis. Support Syst..

[52]  Bruce E. Wampold,et al.  The Intellectual Foundations of Education , 2009 .

[53]  Josephine E. Olson,et al.  Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields , 2005, Interfaces.

[54]  N. Adler,et al.  When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings , 2009 .

[55]  Mark L. Gillenson,et al.  Journal Rakings 2008: A Synthesis of Studies , 2008, AMCIS.

[56]  Tove Faber Frandsen,et al.  Journal self-citations - Analysing the JIF mechanism , 2007, J. Informetrics.

[57]  John Mingers,et al.  Ranking journals in business and management: a statistical analysis of the Harzing data set , 2007, Eur. J. Inf. Syst..

[58]  Grant Lewison.  Researchers" and users" perceptions of the relative standing of biomedical papers in different journals , 2004, Scientometrics.

[59]  R. Rousseau.  Temporal differences in self-citation rates of scientific journals , 2006, Scientometrics.

[60]  Michael D. Gordon,et al.  Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences , 1982, J. Am. Soc. Inf. Sci..

[61]  Paul Genoni,et al.  Citation analysis and peer ranking of Australian social science journals , 2010, Scientometrics.

[62]  Mark L. Gillenson,et al.  Academic Issues in MIS: Journals and Books , 1991, MIS Q..

[63]  J V Greer.  Quality is in the Eye of the Beholder , 1991, Exceptional children.

[64]  R. Coe,et al.  Evaluating the management journals: a second look. , 1984, Academy of Management journal. Academy of Management.

[65]  A Routh,et al.  A Case of Cæsarean Hysterectomy for Traumatic Atresia of the Vagina, the Patient Having Previously Undergone a Successful Operation for a Vesico-Vaginal Fistula Due to the Same Injury , 1908, Proceedings of the Royal Society of Medicine.

[66]  Nick Bontis,et al.  A follow-up ranking of academic journals , 2009, J. Knowl. Manag..

[67]  Rajiv Kohli,et al.  An AHP analysis of quality in AI and DSS journals , 2002 .

[68]  Hans H. K. Sønderstrup-Andersen,et al.  An investigation into diabetes researcher’s perceptions of the Journal Impact Factor — reconsidering evaluating research , 2008, Scientometrics.

[69]  Jeffrey R. Lacasse,et al.  Evaluating Journal Quality: Is the H-Index a Better Measure Than Impact Factors? , 2011 .

[70]  Alexander Serenko,et al.  The Development of an AI Journal Ranking List Based on the Revealed Preference Approach , 2010, AMCIS.

[71]  D. Christakis,et al.  Impact factor: a valid measure of journal quality? , 2003, Journal of the Medical Library Association : JMLA.

[72]  Lawrence Feick,et al.  Publish and Prosper: The Financial Impact of Publishing by Marketing Faculty , 2008, Mark. Sci..

[73]  Clyde W. Holsapple,et al.  The impact of periodicals on expert systems research , 1994, IEEE Expert.

[74]  Jeremy B. Fox,et al.  A multi-method evaluation of journals in the decision and management sciences by US academics , 2000 .

[75]  F. DuBois,et al.  Ranking the International Business Journals , 2000 .

[76]  Dominic Upton,et al.  Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK , 2009, Scientometrics.

[77]  Clyde W. Holsapple,et al.  Citation-Based Journal Rankings for AI Research , 1996 .

[78]  Sandra Rousseau,et al.  Journal evaluation by environmental and resource economists: A survey , 2007, Scientometrics.

[79]  John McCarthy,et al.  Programs with common sense , 1960 .