The Distribution of the Asymptotic Number of Citations to Sets of Publications by a Researcher or from an Academic Department Are Consistent with a Discrete Lognormal Model

How to quantify the impact of a researcher’s or an institution’s body of work is a matter of increasing importance to scientists, funding agencies, and hiring committees. The use of bibliometric indicators, such as the h-index or the Journal Impact Factor, have become widespread despite their known limitations. We argue that most existing bibliometric indicators are inconsistent, biased, and, worst of all, susceptible to manipulation. Here, we pursue a principled approach to the development of an indicator to quantify the scientific impact of both individual researchers and research institutions grounded on the functional form of the distribution of the asymptotic number of citations. We validate our approach using the publication records of 1,283 researchers from seven scientific and engineering disciplines and the chemistry departments at the 106 U.S. research institutions classified as “very high research activity”. Our approach has three distinct advantages. First, it accurately captures the overall scientific impact of researchers at all career stages, as measured by asymptotic citation counts. Second, unlike other measures, our indicator is resistant to manipulation and rewards publication quality over quantity. Third, our approach captures the time-evolution of the scientific impact of research institutions.

[1]  Matthew J. Salganik,et al.  Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market , 2006, Science.

[2]  Peter Vinkler,et al.  Characterization of the impact of sets of scientific papers: The Garfield (impact) factor , 2004, J. Assoc. Inf. Sci. Technol..

[3]  Suzie Allard,et al.  Trust and Authority in Scholarly Communications in the Light of the Digital Transition: setting the scene for a major study , 2014, Learn. Publ..

[4]  Jonathan Furner,et al.  Scholarly communication and bibliometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[5]  Marta Sales-Pardo,et al.  Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal , 2010, J. Assoc. Inf. Sci. Technol..

[6]  Martin Rosvall,et al.  Maps of random walks on complex networks reveal community structure , 2007, Proceedings of the National Academy of Sciences.

[7]  Philip M. Davis,et al.  eJournal interface can influence usage statistics: Implications for libraries, publishers, and Project COUNTER , 2006, J. Assoc. Inf. Sci. Technol..

[8]  Richard Van Noorden,et al.  The top 100 papers , 2014, Nature.

[9]  Adrian Letchford,et al.  The advantage of short paper titles , 2015, Royal Society Open Science.

[10]  Leo Egghe,et al.  Dynamic h-index: The Hirsch index in function of time , 2007, J. Assoc. Inf. Sci. Technol..

[11]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[12]  Isabelle Boutron,et al.  Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study , 2012, PLoS medicine.

[13]  Quentin L. Burrell,et al.  Stochastic modelling of the first-citation distribution , 2004, Scientometrics.

[14]  Filippo Radicchi,et al.  The Possible Role of Resource Requirements and Academic Career-Choice Risk on Gender Differences in Publication Rate and Impact , 2012, PloS one.

[15]  H. Stanley,et al.  Methods for measuring the citations and productivity of scientists across time and discipline. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  Amin Mazloumian,et al.  Predicting Scholars' Scientific Impact , 2012, PloS one.

[17]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[18]  J. Bohannon Who's afraid of peer review? , 2013, Science.

[19]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[20]  Francis Narin,et al.  Bibliometric performance measures , 1996, Scientometrics.

[21]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[22]  William Shockley,et al.  On the Statistics of Individual Variations of Productivity in Research Laboratories , 1957, Proceedings of the IRE.

[23]  Konrad Paul Kording,et al.  Future impact: Predicting scientific success , 2012, Nature.

[24]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[25]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[26]  Quentin L. Burrell,et al.  Predicting future citation behavior , 2003, J. Assoc. Inf. Sci. Technol..

[27]  Laura Bonetta,et al.  Should You Be Tweeting? , 2009, Cell.

[28]  Leif Engqvist,et al.  The h-index and self-citations. , 2008, Trends in ecology & evolution.

[29]  Richard Van Noorden,et al.  Metrics: Do metrics matter? , 2010, Nature.

[30]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[31]  S. Fortunato,et al.  Statistical physics of social dynamics , 2007, 0710.3256.

[32]  A. D. Jackson,et al.  Measures for measures , 2006, Nature.

[33]  Michael Schreiber,et al.  Self-citation corrections for the Hirsch index , 2007 .

[34]  Francisco Herrera,et al.  h-Index: A review focused in its variants, computation and standardization for different scientific fields , 2009, J. Informetrics.

[35]  Declan Butler,et al.  Investigating journals: The dark side of publishing , 2013, Nature.

[36]  M. Sales-Pardo,et al.  Effectiveness of Journal Ranking Schemes as a Tool for Locating Information , 2008, PloS one.

[37]  Santo Fortunato,et al.  On the Predictability of Future Impact in Science , 2013, Scientific Reports.

[38]  S. Redner Citation statistics from 110 years of physical review , 2005, physics/0506056.

[39]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[40]  Lutz Bornmann,et al.  What do we know about the h index? , 2007, J. Assoc. Inf. Sci. Technol..

[41]  Terrence A. Brooks,et al.  Evidence of complex citer motivations , 1986, J. Am. Soc. Inf. Sci..

[42]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[43]  Henk F. Moed,et al.  Journal impact measures in bibliometric research , 2004, Scientometrics.

[44]  Santo Fortunato,et al.  Diffusion of scientific credits and the ranking of scientists , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[45]  Fiorenzo Franceschini,et al.  Analysis of the ch-index: an indicator to evaluate the diffusion of scientific research output by citers , 2010, Scientometrics.

[46]  Roberta Kwok,et al.  Research impact: Altmetrics make their mark , 2013, Nature.

[47]  Sibele Fausto,et al.  Blogging : Indexing and Registering the Change in Science 2 . 0 , 2012 .

[48]  E. Fong,et al.  Coercive Citation in Academic Publishing , 2012, Science.

[49]  Sauro Succi,et al.  Statistical regularities in the rank-citation profile of scientists , 2011, Scientific reports.

[50]  E. Garfield The history and meaning of the journal impact factor. , 2006, JAMA.