Scholarly impact assessment: a survey of citation weighting solutions

AbstractScholarly impact assessment has always been a hot issue. It has played an important role in evaluating researchers, scientific papers, scientific teams, and institutions within science of science. Scholarly impact assessment is also used to address fundamental issues, such as reward evaluation, funding allocation, promotion and recruitment decision. Scholars generally agree that it is more reasonable to use weighted citations to assess the scholarly impact. Although a great number of researchers use weighted citations to access the scholarly impact, there is a lack of a systematic summary of citation weighting methods. To fill the gap, this paper summarizes the existing classical indicators and weighting methods used in measuring scholarly impact from the perspectives of articles, authors and journals. We also summarize the focus of the indicators involved in this paper and the weighting factors that involved in the weighting methods. Finally, we discuss the open issues to try to discover the hidden trends of citation weighting. Through this paper, we can not only have a clearer understanding of the weighting methods in the scholarly impact assessment, but also think more deeply about the weighting factors to be explored.

[1]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[2]  Peng Dong,et al.  The "impact factor" revisited , 2005, Biomedical digital libraries.

[3]  Philip M. Davis Eigenfactor : Does the Principle of Repeated Improvement Result in Better Journal Impact Estimates than Raw Citation Counts? , 2008, J. Assoc. Inf. Sci. Technol..

[4]  Michael Schreiber,et al.  To share the fame in a fair way, hm modifies h for multi-authored manuscripts , 2008 .

[5]  Massimo Franceschet,et al.  TimeRank: A dynamic approach to rate scholars using citations , 2017, J. Informetrics.

[6]  Yang Tao,et al.  A Study on Development Planning for Management Science and Engineering , 2006 .

[7]  Wolfgang Glänzel,et al.  A Hirsch-type index for journals , 2006, Scientometrics.

[8]  Qiang Wu,et al.  The w-index: A measure to assess scientific impact by focusing on widely cited papers , 2010, J. Assoc. Inf. Sci. Technol..

[9]  Ludo Waltman,et al.  A new methodology for constructing a publication-level classification system of science , 2012, J. Assoc. Inf. Sci. Technol..

[10]  Feng Xia,et al.  An Overview on Evaluating and Predicting Scholarly Article Impact , 2017, Inf..

[11]  H. Moed CWTS crown indicator measures citation impact of a research group's publication oeuvre , 2010, J. Informetrics.

[12]  Ludo Waltman,et al.  Field-normalized citation impact indicators using algorithmically constructed classification systems of science , 2015, J. Informetrics.

[13]  E. Garfield The history and meaning of the journal impact factor. , 2006, JAMA.

[14]  Dragomir R. Radev,et al.  NLP-driven citation analysis for scientometrics , 2016, Natural Language Engineering.

[15]  Gad Saad,et al.  Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively , 2006, Scientometrics.

[16]  Ying Ding,et al.  Discovering author impact: A PageRank perspective , 2010, Inf. Process. Manag..

[17]  Hong Guo,et al.  PrestigeRank: A new evaluation method for papers and journals , 2011, J. Informetrics.

[18]  Chun-Ting Zhang,et al.  A proposal for calculating weighted citations based on author rank , 2009, EMBO reports.

[19]  Johan Bollen,et al.  A Principal Component Analysis of 39 Scientific Impact Measures , 2009, PloS one.

[20]  Hongyuan Zha,et al.  Co-ranking Authors and Documents in a Heterogeneous Network , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[21]  Karel Jezek,et al.  Author ranking based on personalized PageRank , 2015, J. Informetrics.

[22]  Feng Xia,et al.  Identifying Anomalous Citations for Objective Evaluation of Scholarly Article Impact , 2016, PloS one.

[23]  Dejan Pajic,et al.  On the stability of citation-based journal rankings , 2015, J. Informetrics.

[24]  Carl T. Bergstrom,et al.  The Eigenfactor MetricsTM: A Network Approach to Assessing Scholarly Journals , 2010, Coll. Res. Libr..

[25]  Feng Xia,et al.  Evaluating the Impact of Articles with Geographical Distances between Institutions , 2017, WWW.

[26]  Carl T. Bergstrom,et al.  Author-level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community , 2013, J. Assoc. Inf. Sci. Technol..

[27]  Xiaoming Zhang,et al.  Future Influence Ranking of Scientific Literature , 2014, SDM.

[28]  Michel Zitt,et al.  Citing-side normalization of journal impact: A robust variant of the Audience Factor , 2010, J. Informetrics.

[29]  Ricardo B. C. Prudêncio,et al.  Good to be Bad? Distinguishing between Positive and Negative Citations in Scientific Impact , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[30]  Vladimir Batagelj,et al.  Optimizing SCImago Journal & Country Rank classification by community detection , 2014, J. Informetrics.

[31]  Jonas Lundberg,et al.  Lifting the crown - citation z-score , 2007, J. Informetrics.

[32]  Johan Bollen,et al.  Refining dermatology journal impact factors using PageRank. , 2007, Journal of the American Academy of Dermatology.

[33]  Mônica G. Campiteli,et al.  An index to quantify an individual's scientific research valid across disciplines , 2005 .

[34]  Dalibor Fiala,et al.  Time-aware PageRank for bibliographic networks , 2012, J. Informetrics.

[35]  Massimo Franceschet,et al.  The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis , 2010, J. Informetrics.

[36]  Jiaying Liu,et al.  VOPRec: Vector Representation Learning of Papers with Text Information and Structural Identity for Recommendation , 2021, IEEE Transactions on Emerging Topics in Computing.

[37]  Elke Lang,et al.  Do we need a Unique Scientist ID for publications in biomedicine? , 2005, Biomedical digital libraries.

[38]  Carl T. Bergstrom,et al.  Static Ranking of Scholarly Papers using Article-Level Eigenfactor (ALEF) , 2016, ArXiv.

[39]  Khaled Moustafa,et al.  The Disaster of the Impact Factor , 2015, Sci. Eng. Ethics.

[40]  Karol Zyczkowski,et al.  Citation graph, weighted impact factors and performance indices , 2009, Scientometrics.

[41]  Farrokh Habibzadeh,et al.  Journal weighted impact factor: A proposal , 2008, J. Informetrics.

[42]  Ramin Sadeghi,et al.  Comparison Between Impact Factor, Eigenfactor Metrics, and SCimago Journal Rank Indicator of Pediatric Neurology Journals , 2014, Acta informatica medica : AIM : journal of the Society for Medical Informatics of Bosnia & Herzegovina : casopis Drustva za medicinsku informatiku BiH.

[43]  Liying Yang,et al.  Evaluating journal quality: A review of journal citation indicators and ranking in business and management , 2016, Eur. J. Oper. Res..

[44]  Oren Etzioni,et al.  Identifying Meaningful Citations , 2015, AAAI Workshop: Scholarly Big Data.

[45]  Tam Cam Ha,et al.  The journal impact factor: too much of an impact? , 2006, Annals of the Academy of Medicine, Singapore.

[46]  Nils T. Hagen,et al.  Harmonic publication and citation counting: sharing authorship credit equitably – not equally, geometrically or arithmetically , 2009, Scientometrics.

[47]  M. Kosmulski A new Hirsch-type index saves time and works equally well as the original h-index , 2009 .

[48]  Bradley M. Hemminger,et al.  Scientometrics 2.0: New metrics of scholarly impact on the social Web , 2010, First Monday.

[49]  A. Casadevall,et al.  Causes for the Persistence of Impact Factor Mania , 2014, mBio.

[50]  Robin K. S. Hankin,et al.  Beyond the Durfee square: Enhancing the h-index to score total publication output , 2008, Scientometrics.

[51]  Alan Fersht,et al.  The most influential journals: Impact Factor and Eigenfactor , 2009, Proceedings of the National Academy of Sciences.

[52]  Lawrence J. Smolinsky Expected number of citations and the crown indicator , 2016, J. Informetrics.

[53]  L. Egghe An improvement of the h-index: the g-index , 2006 .

[54]  Ludo Waltman,et al.  Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison , 2012, Scientometrics.

[55]  François Rousselot,et al.  PageRank for bibliographic networks , 2008, Scientometrics.

[56]  Xiaojun Wan,et al.  Are all literature citations equally important? Automatic citation strength estimation and its applications , 2014, J. Assoc. Inf. Sci. Technol..

[57]  L. Bornmann,et al.  The journal Impact Factor and alternative metrics , 2016, EMBO reports.

[58]  Ewen Callaway,et al.  Beat it, impact factor! Publishing elite turns against controversial metric , 2016, Nature.

[59]  Shuai Ma,et al.  Ensemble Enabled Weighted PageRank , 2016, ArXiv.

[60]  Ming Zeng,et al.  Ranking Scientific Articles by Exploiting Citations, Authors, Journals, and Time Information , 2013, AAAI.

[61]  Dragomir R. Radev,et al.  Purpose and Polarity of Citation: Towards NLP-based Bibliometrics , 2013, NAACL.

[62]  Stefano Bertuzzi,et al.  ASM Journals Eliminate Impact Factor Information from Journal Websites , 2016, Infection and Immunity.

[63]  Anne-Wil Harzing,et al.  Changing of the guard , 2009 .

[64]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[65]  K. Fujita,et al.  Detecting research fronts using different types of weighted citation networks , 2012, 2012 Proceedings of PICMET '12: Technology Management for Emerging Technologies.

[66]  Ronald Rousseau,et al.  Journal of Informetrics -index , 2022 .

[67]  Lise Getoor,et al.  FutureRank: Ranking Scientific Articles by Predicting their Future PageRank , 2009, SDM.

[68]  Serge Galam,et al.  Tailor based allocations for multiple authorship: a fractional gh-index , 2010, Scientometrics.

[69]  E. Roldán-Valadez,et al.  Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: a linear mixed design model analysis of Radiology, Nuclear Medicine and Medical Imaging journals , 2018, La radiologia medica.

[70]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[71]  Richard J. C. Brown A simple method for excluding self-citation from the h-index: the b-index , 2009, Online Inf. Rev..

[72]  Enrique Orduña-Malea,et al.  Google Scholar Metrics evolution: an analysis according to languages , 2013, Scientometrics.

[73]  Ludo Waltman,et al.  Generalizing the H- and G-Indices , 2008, J. Informetrics.

[74]  Ying Ding,et al.  Measuring scholarly impact in heterogeneous networks , 2010, ASIST.

[75]  C. Michael Hall,et al.  Following the impact factor: Utilitarianism or academic compliance? , 2015 .

[76]  S. Ramin,et al.  Comparison between Impact factor, SCImago journal rank indicator and Eigenfactor score of nuclear medicine journals. , 2012, Nuclear medicine review. Central & Eastern Europe.

[77]  Michel Zitt,et al.  Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation , 2005, Scientometrics.

[78]  David B Elliott,et al.  The impact factor: a useful indicator of journal quality or fatally flawed? , 2014, Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians.

[79]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[80]  Francisco Herrera,et al.  hg-index: a new index to characterize the scientific output of researchers based on the h- and g-indices , 2010, Scientometrics.

[81]  Mônica G. Campiteli,et al.  Is it possible to compare researchers with different scientific interests? , 2006, Scientometrics.

[82]  Daniel Lemire,et al.  Measuring academic influence: Not all citations are equal , 2015, J. Assoc. Inf. Sci. Technol..

[83]  Chun-Ting Zhang,et al.  The e-Index, Complementing the h-Index for Excess Citations , 2009, PloS one.

[84]  Loet Leydesdorff,et al.  of Science , 2022 .

[85]  Vijay Prakash Mathur,et al.  Impact factor and other standardized measures of journal citation: a perspective. , 2009, Indian journal of dental research : official publication of Indian Society for Dental Research.

[86]  Rodrigo Costas,et al.  The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level , 2007, J. Informetrics.

[87]  Sergei Maslov,et al.  Ranking scientific publications using a model of network traffic , 2006, ArXiv.

[88]  Wolfgang Glänzel,et al.  The weight of author self-citations. A fractional approach to self-citation counting , 2006, Scientometrics.

[89]  Loet Leydesdorff,et al.  Normalization at the field level: fractional counting of citations , 2010, J. Informetrics.

[90]  Jordi Comas,et al.  Developing Virtual Worlds: The Interplay of Design, Communities and Rationality , 2010, First Monday.

[91]  Dalibor Fiala Current index: A Proposal for a dynamic rating system for researchers , 2014, J. Assoc. Inf. Sci. Technol..

[92]  Ludo Waltman,et al.  A systematic empirical comparison of different approaches for normalizing citation impact indicators , 2013, J. Informetrics.

[93]  Michel Zitt,et al.  Modifying the journal impact factor by fractional citation weighting: The audience factor , 2008, J. Assoc. Inf. Sci. Technol..

[94]  Ignacio Roa,et al.  Comparison between Impact Factor, Eigenfactor Score, and SCImago Journal Rank Indicator in Anatomy and Morphology Journals Comparación entre Factor de Impacto, Eigenfactor Score e Indicador SCImago Journal Rank en Revistas de Anatomía y Morfología , 2015 .

[95]  Feng Xia,et al.  PNCOIRank: Evaluating the Impact of Scholarly Articles with Positive and Negative Citations , 2016, WWW.

[96]  Loet Leydesdorff,et al.  Remaining problems with the "New Crown Indicator" (MNCS) of the CWTS , 2010, J. Informetrics.

[97]  Carl T. Bergstrom Eigenfactor Measuring the value and prestige of scholarly journals , 2007 .

[98]  Vicente P. Guerrero-Bote,et al.  A new approach to the metric of journals' scientific prestige: The SJR indicator , 2010, J. Informetrics.

[99]  Ludo Waltman,et al.  The relation between Eigenfactor, audience factor, and influence weight , 2010 .

[100]  Dalibor Fiala,et al.  PageRank variants in the evaluation of citation networks , 2014, J. Informetrics.

[101]  Matús Medo,et al.  Quantifying and suppressing ranking bias in a large citation network , 2017, J. Informetrics.

[102]  Michael Schreiber,et al.  The influence of self‐citation corrections and the fractionalised counting of multi‐authored manuscripts on the Hirsch index , 2009 .

[103]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[104]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[105]  Thed N. van Leeuwen,et al.  Some modifications to the SNIP journal impact indicator , 2012, J. Informetrics.

[106]  Ludo Waltman,et al.  A recursive field-normalized bibliometric performance indicator: an application to the field of library and information science , 2011, Scientometrics.

[107]  Katsuhide Fujita,et al.  Detecting Research Fronts Using Neural Network Model for Weighted Citation Network Analysis , 2015, J. Inf. Process..

[108]  Ç. Şekercioğlu Quantifying Coauthor Contributions , 2008, Science.

[109]  Ludo Waltman,et al.  Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie , 2008, J. Informetrics.

[110]  Henk F. Moed,et al.  Measuring contextual citation impact of scientific journals , 2009, J. Informetrics.

[111]  C. Zuiani,et al.  Automated breast ultrasound: basic principles and emerging clinical applications , 2017, La radiologia medica.

[112]  Johan Bollen,et al.  Journal status , 2006, Scientometrics.

[113]  D. Snead,et al.  Medical Students’ Perceptions of Clinical Teachers as Role Model , 2016, PloS one.