Mendeley readership altmetrics for medical articles: An analysis of 45 fields

Medical research is highly funded and often expensive and so is particularly important to evaluate effectively. Nevertheless, citation counts may accrue too slowly for use in some formal and informal evaluations. It is therefore important to investigate whether alternative metrics could be used as substitutes. This article assesses whether one such altmetric, Mendeley readership counts, correlates strongly with citation counts across all medical fields, whether the relationship is stronger if student readers are excluded, and whether they are distributed similarly to citation counts. Based on a sample of 332,975 articles from 2009 in 45 medical fields in Scopus, citation counts correlated strongly (about 0.7; 78% of articles had at least one reader) with Mendeley readership counts (from the new version 1 applications programming interface [API]) in almost all fields, with one minor exception, and the correlations tended to decrease slightly when student readers were excluded. Readership followed either a lognormal or a hooked power law distribution, whereas citations always followed a hooked power law, showing that the two may have underlying differences.

[1]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[2]  Vincent Larivière,et al.  Tweets vs. Mendeley readers: How do these two social media metrics differ? , 2014, it Inf. Technol..

[3]  Q. Vuong Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[4]  J. Hemphill,et al.  Interpreting the magnitudes of correlation coefficients. , 2003, The American psychologist.

[5]  Liz Allen,et al.  Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation , 2012, BMJ Open.

[6]  Loet Leydesdorff,et al.  The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 , 2012, J. Informetrics.

[7]  D. Wardle Do 'Faculty of 1000' (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? , 2010 .

[8]  Mike Thelwall,et al.  Do highly cited researchers successfully use the social web? , 2014, Scientometrics.

[9]  Jenny Fry,et al.  Measuring researchers’ use of scholarly information through social bookmarking data: A case study of BibSonomy , 2012, J. Inf. Sci..

[10]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[11]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[12]  Massimo Franceschet,et al.  The first Italian research assessment exercise: A bibliometric perspective , 2009, J. Informetrics.

[13]  Mark E. J. Newman,et al.  Power-Law Distributions in Empirical Data , 2007, SIAM Rev..

[14]  S. S. Wilks The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses , 1938 .

[15]  Ehsan Mohammadi,et al.  Identifying the invisible impact of scholarly publications : a multi-disciplinary analysis using 'altmetrics' , 2015 .

[16]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[17]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.

[18]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[19]  Mike Thelwall,et al.  Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis , 2007, J. Assoc. Inf. Sci. Technol..

[20]  Daryl E. Chubin,et al.  Content Analysis of References: Adjunct or Alternative to Citation Counting? , 1975 .

[21]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[22]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[23]  Stefanie Haustein,et al.  Exploring data quality and retrieval strategies for mendeley reader counts , 2015 .

[24]  Mike Thelwall,et al.  Tweeting Links to Academic Articles , 2013 .

[25]  David M. Pennock,et al.  Winners don't take all: Characterizing the competition for links on the web , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[27]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[28]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[29]  M. Thelwall,et al.  Policy-relevant Webometrics for individual scientific fields , 2010 .

[30]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[31]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[32]  Paul Wilson,et al.  The misuse of the Vuong test for non-nested models to test for zero-inflation , 2015 .

[33]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[34]  Lada A. Adamic,et al.  Power-Law Distribution of the World Wide Web , 2000, Science.

[35]  Heather A. Piwowar,et al.  The power of altmetrics on a CV , 2013 .

[36]  Richard E. West,et al.  Mendeley: Creating Communities of Scholarly Inquiry Through Research Collaboration , 2011 .

[37]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[38]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[39]  G. Lewison Gastroenterology research in the United Kingdom: funding sources and impact , 1998, Gut.

[40]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[41]  Mike Thelwall,et al.  Distributions for cited articles from individual subjects and years , 2014, J. Informetrics.

[42]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[43]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[44]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[45]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[46]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[47]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[48]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[49]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[50]  Stefanie N. Lindstaedt,et al.  Visualization of Co-Readership Patterns from an Online Reference Management System , 2014, J. Informetrics.

[51]  Rodrigo Costas,et al.  What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact? PLoS ALM Workshop, San Francisco , 2013 .

[52]  H. Moed Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[53]  R. Topel,et al.  Diminishing Returns?: The Costs and Benefits of Improving Health , 2003 .

[54]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[55]  Mike Thelwall,et al.  Guideline references and academic citations as evidence of the clinical value of health research , 2016, J. Assoc. Inf. Sci. Technol..

[56]  Victor Henning,et al.  Mendeley - A Last.fm For Research? , 2008, 2008 IEEE Fourth International Conference on eScience.

[57]  Tamara Heck,et al.  Testing Collaborative Filtering against Co-Citation Analysis and Bibliographic Coupling for Academic Author Recommendation , 2011 .

[58]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[59]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..