F1000Prime recommended articles and their citations: an exploratory study of four journals

This study examined F1000Prime recommended research and review articles published in Cell, JAMA: The Journal of the American Medical Association, The Lancet, and The New England Journal of Medicine (NEJM) in 2010. The analyses included (1) the classifications assigned to the articles; (2) differences in Web of Science (WoS) citation counts over 9 years between the articles with F1000Prime recommendations and the other articles of the same journal; (3) correlations between the F1000Prime rating scores and WoS citation counts; (4) scaled graphic comparisons of the two measures; (5) content analysis of the top 5 WoS cited and top 5 F1000Prime scored NEJM articles. The results show that most of the recommended articles were classified as New Finding, Clinical Trial, Conformation, Interesting Hypothesis, and Technical Advance. The top classifications differred between the medical journals (JAMA, The Lancet, and NEJM) and the biology journal (Cell); for the latter, both New Finding and Interesting Hypothesis occurred more frequently than the three medical journals. The articles recommended by F1000 Faculty members were cited significantly more than other articles of the same journal for the three medical journals, but no significance was found between the two sets of articles in Cell. The correlations between the F1000Prime rating scores and WoS citation counts of the articles in the same journal were significant for the two medical journals (The Lancet and NEJM) and the biology journal (Cell). NEJM showed significances in both the upper quantile (top 50%), and the upper quartile (top 25%) sets. One of the medical journals, JAMA, did not show any significant correlation between the two measures. Despite the significant correlations of the three journals, Min–Max scaled graphic comparisons of the two measures did not reveal any patterns for predicting citation trends by F1000Prime rating scores. The peak citation year of the articles ranged from 2 to 8 years after the publication year for NEJM. Content analysis of the top-cited and top-scored NEJM articles found that highly commendable papers with comments such as “exceptional,” “landmark study,” or “paradigm shift” received varied rating scores. In comparison, some of the results corroborate with previous studies. Further studies are suggested to include additional journals and different years as well as alternative methods. Studies are needed to understand how F1000 Faculty assign ratings and what criteria they use. In addition, it is also worth investigating how F1000Prime users perceive the meanings of the ratings.

[1]  D. Benos,et al.  The ups and downs of peer review. , 2007, Advances in physiology education.

[2]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[3]  Lutz Bornmann,et al.  Does quality and content matter for citedness? A comparison with para-textual factors and over time , 2015, J. Informetrics.

[4]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[5]  G. Cumming,et al.  High Impact = High Statistical Standards? Not Necessarily So , 2013, PloS one.

[6]  D. Wardle.  Do 'Faculty of 1000' (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? , 2010 .

[7]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[8]  Lutz Bornmann,et al.  Field- and time-normalization of data with many zeros: an empirical analysis using citation and Twitter data , 2017, Scientometrics.

[9]  Ann Blandford,et al.  Keeping up to date: An academic researcher's information journey , 2017, J. Assoc. Inf. Sci. Technol..

[10]  David P. Doane,et al.  Measuring Skewness: A Forgotten Statistic? , 2011 .

[11]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, Journal of the American Society for Information Science.

[12]  Stefan Hirschauer.  How Editors Decide. Oral Communication in Journal Peer Review , 2015 .

[13]  Lutz Bornmann,et al.  Relative Citation Ratio (RCR): An empirical attempt to study a new field‐normalized bibliometric indicator , 2015, J. Assoc. Inf. Sci. Technol..

[14]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, J. Am. Soc. Inf. Sci..

[15]  Lutz Bornmann,et al.  Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts? , 2014, J. Informetrics.

[16]  Lutz Bornmann,et al.  Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data , 2017, PloS one.

[17]  Henry G. Small,et al.  Why authors think their papers are highly cited , 2004, Scientometrics.

[18]  M. Walport,et al.  Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs , 2009, PloS one.

[19]  Bertil F. Dorch.  On the Citation Advantage of linking to data , 2012 .

[20]  Lutz Bornmann,et al.  Which people use which scientific papers? An evaluation of data from F1000 and Mendeley , 2015, J. Informetrics.

[21]  Jian Du,et al.  The effects of research level and article type on the differences between citation metrics and F1000 recommendations , 2015, J. Assoc. Inf. Sci. Technol..

[22]  Derek J. de Solla Price,et al.  Science Since Babylon , 1961 .

[23]  Jian Wang,et al.  Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P100) , 2013, J. Informetrics.

[24]  Lutz Bornmann,et al.  Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references , 2014, J. Assoc. Inf. Sci. Technol..

[25]  Lutz Bornmann,et al.  Interrater reliability and convergent validity of F1000Prime peer review , 2014, J. Assoc. Inf. Sci. Technol..

[26]  Loet Leydesdorff,et al.  The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 , 2012, J. Informetrics.

[27]  Peiling Wang,et al.  Research classics and citation analysis , 2006, ASIST.

[28]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[29]  Jan Velterop,et al.  Post‐publication filtering and evaluation: Faculty of 1000 , 2003, Learn. Publ..