Identification of milestone papers through time-balanced network centrality

Citations between scientific papers and related bibliometric indices, such as the $h$-index for authors and the impact factor for journals, are being increasingly used - often in controversial ways - as quantitative tools for research evaluation. Yet, a fundamental research question remains still open: to which extent do quantitative metrics capture the significance of scientific works? We analyze the network of citations among the $449,935$ papers published by the American Physical Society (APS) journals between 1893 and 2009, and focus on the comparison of metrics built on the citation count with network-based metrics. We contrast five article-level metrics with respect to the rankings that they assign to a set of fundamental papers, called Milestone Letters, carefully selected by the APS editors for "making long-lived contributions to physics, either by announcing significant discoveries, or by initiating new areas of research". A new metric, which combines PageRank centrality with the explicit requirement that paper score is not biased by paper age, is the best-performing metric overall in identifying the Milestone Letters. The lack of time bias in the new metric makes it also possible to use it to compare papers of different age on the same scale. We find that network-based metrics identify the Milestone Letters better than metrics based on the citation count, which suggests that the structure of the citation network contains information that can be used to improve the ranking of scientific publications. The methods and results presented here are relevant for all evolving systems where network centrality metrics are applied, for example the World Wide Web and online social networks. An interactive Web platform where it is possible to view the ranking of the APS papers by rescaled PageRank is available at the address \url{this http URL}.

[1]  Andreas Spitz,et al.  Measuring Long-Term Impact Based on Network Centrality: Unraveling Cinematic Citations , 2014, PloS one.

[2]  Dima Shepelyansky,et al.  Google matrix analysis of directed networks , 2014, ArXiv.

[3]  S. Liebowitz,et al.  Assessing the Relative Impacts of Economic Journals , 1984 .

[4]  Ludo Waltman,et al.  PageRank-Related Methods for Analyzing Citation Networks , 2014 .

[5]  Claudio Castellano,et al.  A Reverse Engineering Approach to the Suppression of Citation Biases Reveals Universal Properties of Citation Distributions , 2012, PloS one.

[6]  Johan Bollen,et al.  Journal status , 2006, Scientometrics.

[7]  F. Reines,et al.  On the Detection of the Free Neutrino , 1953 .

[8]  Sergei Maslov,et al.  Ranking scientific publications using a model of network traffic , 2006, ArXiv.

[9]  James Wilsdon,et al.  We need a measured approach to metrics , 2015, Nature.

[10]  Filippo Menczer,et al.  Universality of scholarly impact metrics , 2013, J. Informetrics.

[11]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[12]  D. King The scientific impact of nations , 2004, Nature.

[13]  P. H. Heckmann Search for neutrinos from the Sun , 1973 .

[14]  J. Hubbard Calculation of Partition Functions , 1959 .

[15]  Park,et al.  Observation of a neutrino burst in coincidence with supernova 1987A in the Large Magellanic Cloud. , 1987, Physical review letters.

[16]  Mark E. J. Newman,et al.  Prediction of highly cited papers , 2013, ArXiv.

[17]  Filippo Menczer,et al.  Quality versus quantity in scientific impact , 2014, J. Informetrics.

[18]  P. Lawrence Lost in publication: how measurement harms science , 2008 .

[19]  Sophie Ahrens,et al.  Recommender Systems , 2012 .

[20]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[21]  M. E. J. Newman,et al.  The first-mover advantage in scientific publication , 2008, 0809.0522.

[22]  An Zeng,et al.  Ranking scientific publications: the effect of nonlinearity , 2014, Scientific Reports.

[23]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[24]  Michael Szell,et al.  A century of physics , 2015, Nature Physics.

[25]  L. Randall,et al.  An Alternative to compactification , 1999, hep-th/9906064.

[26]  Javier Ruiz-Castillo,et al.  The Citation Merit of Scientific Publications , 2012, PloS one.

[27]  Vicente P. Guerrero-Bote,et al.  A new approach to the metric of journals' scientific prestige: The SJR indicator , 2010, J. Informetrics.

[28]  Matúš Medo,et al.  Identification and impact of discoverers in online social systems , 2015, Scientific Reports.

[29]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[30]  Ying Ding,et al.  Applying centrality measures to impact analysis: A coauthorship network analysis , 2009 .

[31]  B. N. Brockhouse,et al.  Lattice Vibrations in Silicon and Germanium , 1959 .

[32]  E. Garfield Citation analysis as a tool in journal evaluation. , 1972, Science.

[33]  John W. Cahn,et al.  Metallic Phase with Long-Range Orientational Order and No Translational Symmetry , 1984 .

[34]  Luís A Nunes Amaral,et al.  Cross-evaluation of metrics to estimate the significance of creative works , 2015, Proceedings of the National Academy of Sciences.

[35]  Jingfeng Xia,et al.  Who publishes in “predatory” journals? , 2015, J. Assoc. Inf. Sci. Technol..

[36]  Hall,et al.  Direct link between microwave and optical frequencies with a 300 THz femtosecond laser comb , 2000, Physical review letters.

[37]  Yoichiro Nambu,et al.  Axial vector current conservation in weak interactions , 1960 .

[38]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[39]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[40]  S. Wu,et al.  Experimental Observation of a Heavy Particle $J$ , 1974 .

[41]  Qing Ke,et al.  Defining and identifying Sleeping Beauties in science , 2015, Proceedings of the National Academy of Sciences.

[42]  K. Goulianos,et al.  Observation of High-Energy Neutrino Reactions and the Existence of Two Kinds of Neutrinos , 1962 .

[43]  Gourab Ghoshal,et al.  Ranking stability and super-stable nodes in complex networks. , 2011, Nature communications.

[44]  Bo-Christer Björk,et al.  Have the “mega-journals” reached the limits to growth? , 2015, PeerJ.

[45]  Ying Ding,et al.  Applying centrality measures to impact analysis: A coauthorship network analysis , 2009, J. Assoc. Inf. Sci. Technol..

[46]  Dalibor Fiala,et al.  PageRank variants in the evaluation of citation networks , 2014, J. Informetrics.

[47]  Reichert,et al.  Phase coherent vacuum-ultraviolet to radio frequency comparison with a mode-locked laser , 2000, Physical review letters.

[48]  Gabriel Pinski,et al.  Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics , 1976, Inf. Process. Manag..

[49]  E. Purcell,et al.  Relaxation Effects in Nuclear Magnetic Resonance Absorption , 1948 .

[50]  Cornell,et al.  Collective Excitations of a Bose-Einstein Condensate in a Dilute Gas. , 1996, Physical review letters.

[51]  Marcel Dunaiski,et al.  Comparing paper ranking algorithms , 2012, SAICSIT '12.

[52]  Sergei Maslov,et al.  Promise and Pitfalls of Extending Google's PageRank Algorithm to Citation Networks , 2008, The Journal of Neuroscience.

[53]  Dennis G. Zill,et al.  Advanced Engineering Mathematics , 2021, Technometrics.

[54]  Marcel Dunaiski,et al.  Evaluating paper and author ranking algorithms using impact and contribution awards , 2016, J. Informetrics.

[55]  Luis W. Alvarez,et al.  Evidence for a T=0 Three-Pion Resonance , 1961 .

[56]  Yi-Cheng Zhang,et al.  Ranking nodes in growing networks: When PageRank fails , 2015, Scientific Reports.

[57]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[58]  Loet Leydesdorff,et al.  Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis , 2010, PloS one.

[59]  C. Peters,et al.  Generation of optical harmonics , 1961 .

[60]  Claudio Castellano,et al.  Rescaling citations of publications in Physics , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[61]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[62]  An Zeng,et al.  Ranking scientific publications with similarity-preferential mechanism , 2015, Scientometrics.

[63]  Binasch,et al.  Enhanced magnetoresistance in layered magnetic structures with antiferromagnetic interlayer exchange. , 1989, Physical review. B, Condensed matter.

[64]  G. Shaviv,et al.  Present Status of the Theoretical Predictions for the ^(37)Cl Solar-Neutrino Experiment , 1968 .

[65]  Marián Boguñá,et al.  Approximating PageRank from In-Degree , 2007, WAW.

[66]  The Super-Kamiokande Collaboration,et al.  Evidence for oscillation of atmospheric neutrinos , 1998, hep-ex/9807003.

[67]  Anthony J Leggett,et al.  Interpretation of Recent Results on He 3 below 3 mK: A New Liquid Phase? , 1972 .

[68]  Phillips,et al.  Observation of atoms laser cooled below the Doppler limit. , 1988, Physical review letters.

[69]  R. C. Allen,et al.  Direct evidence for neutrino flavor transformation from neutral-current interactions in the Sudbury Neutrino Observatory. , 2002, Physical review letters.

[70]  Albert-László Barabási,et al.  Quantifying Long-Term Scientific Impact , 2013, Science.

[71]  Richard Van Noorden,et al.  Metrics: A profusion of measures. , 2010, Nature.

[72]  R. Werner The focus on bibliometrics makes papers less useful , 2015, Nature.

[73]  Association Information , 2000 .

[74]  David F. Gleich,et al.  PageRank beyond the Web , 2014, SIAM Rev..

[75]  E. Purcell,et al.  Resonance Absorption by Nuclear Magnetic Moments in a Solid , 1946 .

[76]  Etienne,et al.  Giant magnetoresistance of (001)Fe/(001)Cr magnetic superlattices. , 1988, Physical review letters.

[77]  Massimo Franceschet,et al.  PageRank , 2010, Commun. ACM.

[78]  Pavel Berkhin,et al.  A Survey on PageRank Computing , 2005, Internet Math..

[79]  Chu,et al.  Three-dimensional viscous confinement and cooling of atoms by resonance radiation pressure. , 1985, Physical review letters.

[80]  Yi-Cheng Zhang,et al.  Identification and modeling of discoverers in online social systems , 2015, ArXiv.

[81]  Giulio Cimini,et al.  The Scientific Competitiveness of Nations , 2014, PloS one.

[82]  C. K. Chiang,et al.  Electrical Conductivity in Doped Polyacetylene. , 1977 .

[83]  Santo Fortunato,et al.  Diffusion of scientific credits and the ranking of scientists , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[84]  Lenar Iskhakov,et al.  Algorithms and Models for the Web Graph , 2018, Lecture Notes in Computer Science.

[85]  R. Pound,et al.  Apparent Weight of Photons , 1960 .

[86]  Claudio Castellano,et al.  Testing the fairness of citation indicators for comparison across scientific domains: The case of fractional citation counts , 2011, J. Informetrics.

[87]  R. Glauber Coherent and incoherent states of the radiation field , 1963 .

[88]  J. Pendry,et al.  Negative refraction makes a perfect lens , 2000, Physical review letters.

[89]  P. Cai,et al.  Reliable Perturbative Results for Strong Interactions ? , 2011 .

[90]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[91]  J. Farine,et al.  Measurement of the rate of νe+d → p+p+e− interactions produced by 8B solar neutrinos at the Sudbury Neutrino Observatory , 2002 .

[92]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[93]  Bascom S. Deaver,et al.  Experimental Evidence for Quantized Flux in Superconducting Cylinders , 1961 .

[94]  S. Redner Citation statistics from 110 years of physical review , 2005, physics/0506056.

[95]  Santo Fortunato,et al.  Attention Decay in Science , 2015, J. Informetrics.

[96]  Daniel Sarewitz,et al.  The pressure to publish pushes down quality , 2016, Nature.

[97]  Dalibor Fiala,et al.  Time-aware PageRank for bibliographic networks , 2012, J. Informetrics.

[98]  A. Kinney National scientific facilities and their science impact on nonbiomedical research , 2007, Proceedings of the National Academy of Sciences.

[99]  Giulio Cimini,et al.  Temporal effects in the growth of networks , 2011, Physical review letters.

[100]  Jean-François Molinari,et al.  A new methodology for ranking scientific institutions , 2008, Scientometrics.

[101]  Mark Newman,et al.  Networks: An Introduction , 2010 .

[102]  R. Giacconi,et al.  Evidence for x Rays From Sources Outside the Solar System , 1962 .

[103]  L. Randall,et al.  A Large mass hierarchy from a small extra dimension , 1999, hep-ph/9905221.

[104]  F. Wilczek,et al.  Ultraviolet Behavior of Non-Abelian Gauge Theories , 1973 .

[105]  Sergei Maslov,et al.  Finding scientific gems with Google's PageRank algorithm , 2006, J. Informetrics.