Identifying “hot papers” and papers with “delayed recognition” in large-scale datasets by using dynamically normalized citation impact scores

Abstract“Hot papers” (HPs) are papers which received a boost of citations shortly after publication. Papers with “delayed recognition” (DRs) received scarcely impact over a long time period, before a considerable citation boost started. DRs have attracted a lot of attention in scientometrics and beyond. Based on a comprehensive dataset with more than 5,000,000 papers published between 1980 and 1990, we identified HPs and DRs. In contrast to many other studies on DRs, which are based on raw citation counts, we calculated dynamically field-normalized impact scores for the search of HPs and DRs. This study is intended to investigate the differences between HPs (n = 323) and DRs (n = 315). The investigation of the journals which have published HPs and DRs revealed that some journals (e.g. Physical Review Letters and PNAS) were able to publish significantly more HPs than other journals. This pattern did not appear in DRs. Many HPs and DRs have been published by authors from the USA; however, in contrast to other countries, authors from the USA have published statistically significantly more HPs than DRs. Whereas “Biochemistry & Molecular Biology,” “Immunology,” and “Cell Biology” have published significantly more HPs than DRs, the opposite result arrived for “Surgery” and “Orthopedics.” The results of the analysis of certain properties of HPs and DRs (e.g. number of pages) suggest that the emergence of DRs is an unpredictable process.

[1]  Vincent Larivière,et al.  What determines researchers’ scientific impact? A case study of Quebec researchers , 2016 .

[2]  Loet Leydesdorff,et al.  Group‐based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims” , 2013, J. Assoc. Inf. Sci. Technol..

[3]  Amber Williams,et al.  Sleeping Beauties of Science. , 2015, Scientific American.

[4]  Lutz Bornmann,et al.  “Smart girls” versus “sleeping beauties” in the sciences: The identification of instant and delayed recognition by using the citation angle , 2016, J. Assoc. Inf. Sci. Technol..

[5]  Krzysztof Zbigniew Stanek.  How long should an astronomical paper be to increase its Impact? , 2008, ArXiv.

[6]  C. Gillmor,et al.  Citation characteristics of the JATP literature , 1975 .

[7]  Donald de B. Beaver,et al.  Does collaborative research have greater epistemic authority? , 2004, Scientometrics.

[8]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[9]  Iman Tahamtan,et al.  Factors affecting number of citations: a comprehensive review of the literature , 2016, Scientometrics.

[10]  Loet Leydesdorff,et al.  Identification of long-term concept-symbols among citations: Can documents be clustered in terms of common intellectual histories? , 2016, ArXiv.

[11]  Qing Ke,et al.  Defining and identifying Sleeping Beauties in science , 2015, Proceedings of the National Academy of Sciences.

[12]  Mike Thelwall,et al.  Determinants of research citation impact in nanoscience and nanotechnology , 2013, J. Assoc. Inf. Sci. Technol..

[13]  Jiang Li,et al.  A study of the "heartbeat spectra" for "sleeping beauties" , 2014, J. Informetrics.

[14]  Jose M Valderas,et al.  Why Do Team-Authored Papers Get Cited More? , 2007, Science.

[15]  E. Garfield,et al.  More Delayed Recognition. Part 2. From Inhibin to Scanning Electron Microscopy , 1990 .

[16]  Jerome K. Vanclay,et al.  Factors affecting citation rates in environmental science , 2013, J. Informetrics.

[17]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[18]  R Alexander Bentley,et al.  Why do team-authored papers get cited more? , 2007, Science.

[19]  J. Koricheva,et al.  What determines the citation frequency of ecological papers? , 2005, Trends in ecology & evolution.

[20]  Tom Tregenza,et al.  Gender bias in the refereeing process , 2002 .

[21]  Anthony F. J. van Raan.  Sleeping beauties cited in patents: Is there also a dormitory of inventions? , 2016, Scientometrics.

[22]  Stephen J. Bensman.  The evaluation of research by scientometric indicators , 2011, J. Assoc. Inf. Sci. Technol..

[23]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[24]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[25]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[26]  Dennis Fok,et al.  Modelling the Diffusion of Scientific Publications , 2005 .

[27]  Gregory D. Webster,et al.  Hot Topics and Popular Papers in Evolutionary Psychology: Analyses of Title Words and Citation Counts in Evolution and Human Behavior, 1979 – 2008 , 2009 .

[28]  A. Raan.  Measuring Science: Capita Selecta of Current Main Issues , 2004 .

[29]  H. Queisser,et al.  Detailed Balance Limit of Efficiency of p‐n Junction Solar Cells , 1961 .

[30]  Jiang Li,et al.  The phenomenon of all-elements-sleeping-beauties in scientific literature , 2012, Scientometrics.

[31]  Ben Van Calster,et al.  It takes time: A remarkable example of delayed recognition , 2012, J. Assoc. Inf. Sci. Technol..

[32]  Rex B. Kline,et al.  Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research , 2004 .

[33]  Tien-Chi Huang,et al.  Systematic Methodology for Excavating Sleeping Beauty Publications and Their Princes from Medical and Biological Engineering Studies , 2015 .

[34]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: an empirical analysis , 2010, Scientometrics.

[35]  Anthony F. J. van Raan,et al.  On Determinants of Citation Scores: A Case Study in Chemical Engineering , 1994, J. Am. Soc. Inf. Sci..

[36]  E. Garfield,et al.  The myth of delayed recognition , 2004 .

[37]  Loet Leydesdorff,et al.  Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data , 2016, J. Informetrics.

[38]  Gunther S. Stent,et al.  Delayed Recognition in Scientific Discovery: Citation Frequency Analysis Aids the Search for Case Histories , 1998 .

[39]  P. Hegarty,et al.  The Consequences of Predicting Scientific Impact in Psychology Using Journal Impact Factors , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[40]  Vincent Larivière,et al.  Long-distance interdisciplinary research leads to higher citation impact , 2014 .

[41]  Daniel Cressey,et al.  ‘Sleeping beauty’ papers slumber for decades , 2015, Nature.

[42]  Jiang Li,et al.  Sleeping beauties in genius work: When were they awakened? , 2016, J. Assoc. Inf. Sci. Technol..

[43]  S. M. Lawani,et al.  Some bibliometric correlates of quality in scientific research , 2005, Scientometrics.

[44]  Fuyuki Yoshikane,et al.  Factors affecting citation rates of research articles , 2015, J. Assoc. Inf. Sci. Technol..

[45]  Cassidy R. Sugimoto,et al.  Bibliometric Standards for Evaluating Research Institutes in the Natural Sciences , 2014 .

[46]  Barbara J. Robson,et al.  Can we predict citation counts of environmental modelling papers? Fourteen bibliographic and categorical variables predict less than 30% of the variability in citation counts , 2016, Environ. Model. Softw..

[47]  Philippe Gorry,et al.  “Sleeping beauty” and her restless sleep: Charles Dotter and the birth of interventional radiology , 2016, Scientometrics.

[48]  Sally Wyatt,et al.  What a difference a colon makes: how superficial factors influence subsequent citation , 2013, Scientometrics.

[49]  Anthony F. J. van Raan,et al.  Bibliometric statistical properties of the 100 largest European research universities: Prevalent scaling rules in the science system , 2008, J. Assoc. Inf. Sci. Technol..

[50]  Alberto Ruano-Ravina,et al.  Evidence-based editing: factors influencing the number of citations in a national journal. , 2012, Annals of epidemiology.

[51]  ANTHONY F. J. VAN RAAN,et al.  Sleeping Beauties in science , 2004, Scientometrics.

[52]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[53]  G. Stent.  Prematurity and uniqueness in scientific discovery. , 1972, Scientific American.

[54]  Thed N. van Leeuwen,et al.  Is scientific literature subject to a 'Sell-By-Date'? A general methodology to analyze the 'durability' of scientific documents , 2009, J. Assoc. Inf. Sci. Technol..

[55]  Wolfgang Glänzel,et al.  Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon , 2004, Scientometrics.

[56]  Isabel Iribarren-Maestro,et al.  Are multi-authorship and visibility related? Study of ten research areas at Carlos III University of Madrid , 2009, Scientometrics.

[57]  C S Peirce,et al.  The numerical measure of the success of predictions. , 1884, Science.

[58]  Werner Marx,et al.  The Shockley‐Queisser paper – A notable example of a scientific sleeping beauty , 2014 .

[59]  A. V. van Raan,et al.  Dormitory of Physical and Engineering Sciences: Sleeping Beauties May Be Sleeping Innovations , 2015, PloS one.

[60]  Tian Yu,et al.  Citation impact prediction for scientific papers using stepwise regression analysis , 2014, Scientometrics.

[61]  Alan C. Acock,et al.  A Gentle Introduction to Stata , 2005 .