Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis

Journal rankings and journal ratings are important to governments, research institutes, and scientific research in general, and they frequently serve as the criteria for evaluating research performance to determine whether specific researchers will receive promotions and/or earn research grants. However, the only widely adopted journal assessment method is known as impact factor (IF), which focuses on citations in academic journals. However, IF disregards the technological applications and value of academic journals. In this article, we propose a method to rank academic journals that utilizes non-patent references in patent documents. We also compare the differences between journal rankings derived by using IF with those derived from the Intellectual Property Citation Index (IPCI) across different fields; moreover, some fields contain positive and significant correlations between IF and the IPCI. The results of this study offer a new perspective from which to assess the technological value of academic journals, particularly those in the technological and scientific fields. This study considers linkages among science and technology and the needs of the stakeholders in journal assessment to shed light on journal assessment and journal ranking methods.

[1]  Daniel G. Bachrach,et al.  The influence of management journals in the 1980s and 1990s , 2005 .

[2]  Francis Narin,et al.  Status report: Linkage between technology and science , 1992 .

[3]  Diana Hicks,et al.  The New York Times as a Resource for Mode 2 , 2013 .

[4]  A L Henley.  A basic list of veterinary medical serials. , 1979, Bulletin of the Medical Library Association.

[5]  Michael John Jones,et al.  Journal evaluation methodologies: A balanced response , 1996 .

[6]  J. P. Haynes,et al.  An empirical method for determining core psychology journals. , 1983 .

[7]  Katherine W. McCain,et al.  Core Journal Networks and Cocitation Maps: New Bibliometric Tools for Serials Research and Management , 1991, The Library Quarterly.

[8]  Michael D. Gordon,et al.  Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences , 1982, J. Am. Soc. Inf. Sci..

[9]  Dominic Upton,et al.  Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK , 2009, Scientometrics.

[10]  Stefano Brusoni,et al.  The knowledge bases of the world’s largest pharmaceutical groups: what do patent citations to non-patent literature reveal? , 2005 .

[11]  Ronald Rousseau,et al.  Journal Evaluation: Technical and Practical Issues, , 2002, Libr. Trends.

[12]  Vijay K. Agrawal.  CONSTITUENCIES OF JOURNALS IN PRODUCTION AND OPERATIONS MANAGEMENT: IMPLICATIONS ON REACH AND QUALITY* , 2009 .

[13]  Koenraad Debackere,et al.  Linking science to technology: Using bibliographic references in patents to build linkage schemes , 2004, Scientometrics.

[14]  Mu-Hsuan Huang,et al.  Technological impact factor: An indicator to measure the impact of academic publications on practical innovation , 2014, J. Informetrics.

[15]  G. Buela-Casal,et al.  ASSESSING THE QUALITY OF ARTICLES AND SCIENTIFIC JOURNALS: PROPOSAL FOR WEIGHTED IMPACT FACTOR AND A QUALITY INDEX , 2004 .

[16]  Jian Ma,et al.  A group decision support approach to evaluating journals , 2004, Inf. Manag..

[17]  D. Pontille,et al.  The controversial policies of journal ratings: evaluating social sciences and humanities , 2010 .

[18]  Shin-Yuan Hung,et al.  Distinct Journal Preference of Successful EC Researchers: A Citation Analysis , 2004 .

[19]  Henk F. Moed,et al.  Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way , 2012, Scientometrics.

[20]  Alan Singleton.  JOURNAL RANKING AND SELECTION: A REVIEW IN PHYSICS , 1976 .

[21]  Anita Coleman,et al.  Assessing the value of a journal beyond the impact factor , 2007, J. Assoc. Inf. Sci. Technol..

[22]  S. Frenkel.  JMP Statistical Discovery Software JMP IN, ver 4 , 2002 .

[23]  P. Pichappan.  A dual refinement of journal self-citation measures , 2005, Scientometrics.

[24]  Peter Vinkler,et al.  Subfield problems in applying the Garfield (Impact) Factors in practice , 2002, Scientometrics.

[25]  Jonathan E. Smith,et al.  The Behavioral Sciences and Management: An Evaluation of Relevant Journals , 1990 .

[26]  Boyd Cohen.  Journal ratings and footprints: a North American perspective of organizations and the natural environment journal quality , 2007 .

[27]  L. Bornmann,et al.  Reference standards and reference multipliers for the comparison of the citation impact of papers published in different time periods , 2010 .

[28]  Francis Narin,et al.  Is technology becoming science? , 1985, Scientometrics.

[29]  Juan Miguel Campanario,et al.  Large increases and decreases in journal impact factors in only one year: The effect of journal self-citations , 2011, J. Assoc. Inf. Sci. Technol..

[30]  Giovanni Abramo,et al.  Mapping Excellence in National Research Systems , 2009, Evaluation review.

[31]  T. Kothari,et al.  Yesterday, today and tomorrow: An overview of research publications in the Journal of International Management , 2012 .

[32]  Kimberly S. Hamilton,et al.  The increasing linkage between U.S. technology and public science , 1997 .

[33]  Bart Van Looy,et al.  Delineating the scientific footprint in technology: Identifying scientific publications within non-patent references , 2011, Scientometrics.

[34]  Rosa Rodriguez-Sánchez,et al.  Overall prestige of journals with ranking score above a given threshold , 2011, Scientometrics.

[35]  Paul Genoni,et al.  Citation analysis and peer ranking of Australian social science journals , 2010, Scientometrics.

[36]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[37]  Carl T. Bergstrom,et al.  Differences in impact factor across fields and over time , 2009 .

[38]  João Claro,et al.  A made-to-measure indicator for cross-disciplinary bibliometric ranking of researchers performance , 2010, Scientometrics.

[39]  R. Wears,et al.  Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. , 2002, JAMA.

[40]  P. Norris,et al.  British and American Journal Evaluation: Divergence or Convergence? * , 1991 .

[41]  G. Duysters,et al.  The technological origins of radical inventions , 2010 .

[42]  Michael Rowlinson,et al.  Journal Rankings in Business and Management and the 2001 Research Assessment Exercise in the UK , 2004 .

[43]  A. Tahai,et al.  A revealed preference study of management journals' direct influences , 1999 .

[44]  Narongrit Sombatsompop,et al.  Making an equality of ISI impact factors for different subject fields , 2005, J. Assoc. Inf. Sci. Technol..

[45]  G. Chapron,et al.  Open, Fair, and Free Journal Ranking for Researchers , 2006 .

[46]  Wolfgang Glänzel,et al.  Journal citation measures: a concise review , 1988, J. Inf. Sci..

[47]  Narongrit Sombatsompop,et al.  Making an equality of ISI impact factors for different subject fields: Research Article , 2005 .

[48]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[49]  Joel A. C. Baum,et al.  Free-Riding on Power Laws: questioning the validity of the Impact Factor as a measure of research quality in organization studies , 2011 .

[50]  Gyula Vastag,et al.  Journal characteristics, rankings and social acculturation in operations management , 2002 .

[51]  Loet Leydesdorff,et al.  Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative research design with policy implications , 2011, J. Assoc. Inf. Sci. Technol..

[52]  David L. Deeds,et al.  An Analysis of the Critical Role of Public Science in Innovation: The Case of Biotechnology , 2000 .

[53]  W. Epstein.  Confirmational Response Bias and the Quality of the Editorial Processes Among American Social Work Journals , 2004 .

[54]  Koenraad Debackere,et al.  Traces of Prior Art: An analysis of non-patent references found in patent documents , 2006, Scientometrics.

[55]  M Sutter,et al.  Tools for Evaluating Research Output , 2001, Evaluation review.

[56]  Making the Connection between Research and Reality: Strategies Teachers Use To Read and Evaluate Journal Articles. , 1997 .

[57]  Szu-chia S. Lo,et al.  Scientific linkage of science research and technology development: a case of genetic engineering research , 2009, Scientometrics.

[58]  Jacques Michel,et al.  Patent citation analysis.A closer look at the basic input data from patent search reports , 2001, Scientometrics.

[59]  Thed N. van Leeuwen,et al.  Technological Relevance of Science: An Assessment of Citation Linkages between Patents and Research Papers , 2000, Scientometrics.

[60]  Martin Meyer,et al.  Tracing knowledge flows in innovation systems , 2002, Scientometrics.

[61]  Mark G. Simkin,et al.  Journal Self-Citation XII: The Ethics of Forced Journal Citations , 2009, Commun. Assoc. Inf. Syst..

[62]  Yen-Liang Chen,et al.  Ranking Taiwanese management journals: A case study , 2008, Scientometrics.

[63]  Xia Gao,et al.  Networks of scientific journals: An exploration of Chinese patent data , 2009, Scientometrics.

[64]  Peter Ingwersen,et al.  Online determination of the journal impact factor and its international properties , 1997, Scientometrics.

[65]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.

[66]  Oscar Volij,et al.  The Measurement of Intellectual Influence , 2002 .

[67]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[68]  Larry M. Robinson,et al.  Measuring the impact of marketing scholars and institutions: an analysis of citation frequency , 1981 .