Web Indicators for Research Evaluation: A Practical Guide

Abstract In recent years there has been an increasing demand for research evaluation within universities and other research-based organisations. In parallel, there has been an increasing recognition that traditional citation-based indicators are not able to reflect the societal impacts of research and are slow to appear. This has led to the creation of new indicators for different types of research impact as well as timelier indicators, mainly derived from the Web. These indicators have been called altmetrics, webometrics or just web metrics. This book describes and evaluates a range of web indicators for aspects of societal or scholarly impact, discusses the theory and practice of using and evaluating web indicators for research assessment and outlines practical strategies for obtaining many web indicators. In addition to describing impact indicators for traditional scholarly outputs, such as journal articles and monographs, it also covers indicators for videos, datasets, software and other non-standard ...

[1]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[2]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[3]  M. Fenner.  What Can Article-Level Metrics Do for You? , 2013, PLoS biology.

[4]  Mike Thelwall,et al.  Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov , 2016, Scientometrics.

[5]  Mike Thelwall,et al.  National research impact indicators from Mendeley readers , 2015, J. Informetrics.

[6]  Loet Leydesdorff,et al.  The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000 , 2012, J. Informetrics.

[7]  A. F. J. VAN RAAN,et al.  In matters of quantitative studies of science the fault of theorists is offering too little and asking too much , 1998, Scientometrics.

[8]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[9]  A. Barrett.  The information-seeking habits of graduate student researchers in the Humanities , 2005 .

[10]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[11]  Andreas Hotho,et al.  Posted, visited, exported: Altmetrics in the social tagging system BibSonomy , 2016, J. Informetrics.

[12]  John P A Ioannidis,et al.  Translation of highly promising basic science research into clinical applications. , 2003, The American journal of medicine.

[13]  Bárbara S. Lancho-Barrantes,et al.  Citation flows in the zones of influence of scientific collaborations , 2012, J. Assoc. Inf. Sci. Technol..

[14]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[15]  Vincent Larivière,et al.  Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter , 2014, J. Assoc. Inf. Sci. Technol..

[16]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[17]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[18]  Mike Thelwall,et al.  How is research blogged? A content analysis approach , 2015, J. Assoc. Inf. Sci. Technol..

[19]  Mike Thelwall,et al.  The discretised lognormal and hooked power law distributions for complete citation data: Best options for modelling and regression , 2016, J. Informetrics.

[20]  Ludo Waltman,et al.  On the calculation of percentile-based bibliometric indicators , 2012, J. Assoc. Inf. Sci. Technol..

[21]  Vincent Larivière,et al.  Tweets vs. Mendeley readers: How do these two social media metrics differ? , 2014, it Inf. Technol..

[22]  K. Hyland,et al.  Disciplinary Discourses, Michigan Classics Ed.: Social Interactions in Academic Writing , 2004 .

[23]  Mark Kats,et al.  Tweeting the Meeting: An In-Depth Analysis of Twitter Activity at Kidney Week 2011 , 2012, PloS one.

[24]  Brian M. Owens.  Research assessments: Judgement day , 2013, Nature.

[25]  Mike Thelwall,et al.  Scholars on soap boxes: Science communication and dissemination in TED videos , 2013, J. Assoc. Inf. Sci. Technol..

[26]  Mike Thelwall,et al.  Are the discretised lognormal and hooked power law distributions plausible for citation data? , 2016, J. Informetrics.

[27]  Stephen S. Murray,et al.  The bibliometric properties of article readership information , 2005, J. Assoc. Inf. Sci. Technol..

[28]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.

[29]  Judit Bar-Ilan,et al.  The Complexity of Measuring the Impact of Books , 2016 .

[30]  Adam Eyre-Walker,et al.  The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations , 2013, PLoS biology.

[31]  C. Wennerås,et al.  Nepotism and sexism in peer-review , 1997, Nature.

[32]  Lutz Bornmann,et al.  Normalization of Mendeley reader impact on the reader- and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader counts , 2016, J. Informetrics.

[33]  Christian Pieter Hoffmann,et al.  A relational altmetric? Network centrality on ResearchGate as an indicator of scientific impact , 2016, J. Assoc. Inf. Sci. Technol..

[34]  Blaise Cronin,et al.  Comparative citation rankings of authors in monographic and journal literature: a study of sociology , 1997, J. Documentation.

[35]  Mike Thelwall,et al.  Can Amazon.com reviews help to assess the wider impacts of books? , 2016, J. Assoc. Inf. Sci. Technol..

[36]  Tim C. E. Engels,et al.  A label for peer-reviewed books , 2013, J. Assoc. Inf. Sci. Technol..

[37]  A. Sharkey,et al.  The Paradox of Publicity , 2014 .

[38]  William Gunn,et al.  Social Signals Reflect Academic Impact: What it Means When a Scholar Adds a Paper to Mendeley , 2013 .

[39]  Dean J. Champion,et al.  A Content Analysis of Book Reviews in the AJS, ASR, and Social Forces , 1973, American Journal of Sociology.

[40]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[41]  Kristina Lerman,et al.  Pragmatic evaluation of folksonomies , 2011, WWW.

[42]  Mike Thelwall,et al.  Assessing non-standard article impact using F1000 labels , 2013, Scientometrics.

[43]  Mike Thelwall,et al.  ResearchGate: Disseminating, communicating, and measuring Scholarship? , 2015, J. Assoc. Inf. Sci. Technol..

[44]  M. Meyer.  Does science push technology? Patents citing scientific literature , 2000 .

[45]  Mike Thelwall,et al.  Alternative metric indicators for funding scheme evaluations , 2016, Aslib J. Inf. Manag..

[46]  Rens Bod,et al.  Can we rank scholarly book publishers? A bibliometric experiment with the field of history , 2015, J. Assoc. Inf. Sci. Technol..

[47]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[48]  E. B. Wilson.  Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[49]  Daqing He,et al.  The correlations between article citation and references' impact measures: What can we learn? , 2013, ASIST.

[50]  Mike Thelwall,et al.  Guideline references and academic citations as evidence of the clinical value of health research , 2016, J. Assoc. Inf. Sci. Technol..

[51]  Thed N. van Leeuwen,et al.  The Leiden ranking 2011/2012: Data collection, indicators, and interpretation , 2012, J. Assoc. Inf. Sci. Technol..

[52]  Wolfgang Glänzel,et al.  A Bibliometric Study of Reference Literature in the Sciences and Social Sciences , 1999, Inf. Process. Manag..

[53]  Richard Van Noorden.  Online collaboration: Scientists and the social network , 2014, Nature.

[54]  R. Procter,et al.  Adoption and use of Web 2.0 in scholarly communications , 2010, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[55]  Christian Zimmermann,et al.  Academic Rankings with RePEc , 2012 .

[56]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[57]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[58]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[59]  Mike Thelwall,et al.  When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals , 2016, J. Assoc. Inf. Sci. Technol..

[60]  Henk F. Moed,et al.  Multidimensional assessment of scholarly research impact , 2014, J. Assoc. Inf. Sci. Technol..

[61]  Nicolás Robinson-García,et al.  Towards a Book Publishers Citation Reports. First approach using the Book Citation Index , 2012, Revista española de Documentación Científica.

[62]  Martyn Poliakoff,et al.  The Periodic Table of Videos , 2011, Science.

[63]  Paul Metz,et al.  A Reputational Study of Academic Publishers. , 1996 .

[64]  Stevan Harnad,et al.  Earlier Web Usage Statistics as Predictors of Later Citation Impact , 2005, J. Assoc. Inf. Sci. Technol..

[65]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1996, Scientometrics.

[66]  Sue Stone,et al.  Humanities scholars: Information Needs and Uses , 1982, J. Documentation.

[67]  A. J. M. Linmans,et al.  Why with bibliometrics the Humanities does not need to be the weakest link , 2010, Scientometrics.

[68]  R. Neumann.  Disciplinary Differences and University Teaching , 2001 .

[69]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[70]  Roberto Cornacchia,et al.  Altmetrics for the humanities: Comparing Goodreads reader ratings with citations to history books , 2015, Aslib J. Inf. Manag..

[71]  Mike Thelwall,et al.  The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach , 2015, J. Informetrics.

[72]  S. Rijcke,et al.  Bibliometrics: The Leiden Manifesto for research metrics , 2015, Nature.

[73]  Daniel Torres-Salinas,et al.  Library Catalog Analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in Economics , 2009, J. Informetrics.

[74]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[75]  Juan Miguel Campanario,et al.  Coverage, field specialisation and the impact of scientific publishers indexed in the Book Citation Index , 2013, Online Inf. Rev..

[76]  Martin Fenner,et al.  Altmetrics in Evolution: Defining and Redefining the Ontology of Article-Level Metrics , 2013 .

[77]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[78]  E. C. Fieller.  SOME PROBLEMS IN INTERVAL ESTIMATION , 1954 .

[79]  James C. Garand,et al.  Ranking Scholarly Publishers in Political Science: An Alternative Approach , 2011, PS: Political Science & Politics.

[80]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[81]  Mike Thelwall,et al.  Interpreting correlations between citation counts and other indicators , 2016, Scientometrics.

[82]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[83]  Wen-Hsiung Wu,et al.  ResearchGate: An effective altmetric indicator for active researchers? , 2016, Comput. Hum. Behav..

[84]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[85]  Mike Thelwall,et al.  Can the impact of scholarly images be assessed online? An exploratory study using image identification technology , 2010 .

[86]  Adam Marcus,et al.  Science publishing: The paper is not sacred , 2011, Nature.

[87]  Johan Bollen,et al.  Usage bibliometrics , 2011, Annu. Rev. Inf. Sci. Technol..

[88]  Kuang-hua Chen,et al.  Exploring alternative metrics of scholarly performance in the social sciences and humanities in Taiwan , 2014, Scientometrics.

[89]  Mike Thelwall,et al.  An automatic method for extracting citations from Google Books , 2015, J. Assoc. Inf. Sci. Technol..

[90]  张静,et al.  Banana Ovate family protein MaOFP1 and MADS-box protein MuMADS1 antagonistically regulated banana fruit ripening , 2015 .

[91]  Liz Allen,et al.  Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact , 2014, PLoS biology.

[92]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[93]  Finn Årup Nielsen,et al.  Scientific citations in Wikipedia , 2007, First Monday.

[94]  Mike Thelwall,et al.  Figshare: a universal repository for academic resource sharing? , 2016, Online Inf. Rev..

[95]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[96]  R. Gray.  Has the Research Excellence Framework killed creativity? , 2015, Journal of psychiatric and mental health nursing.

[97]  Johan Bollen,et al.  Toward alternative metrics of journal impact: A comparison of download and citation data , 2005, Inf. Process. Manag..

[98]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[99]  Thed N. van Leeuwen,et al.  Technological Relevance of Science: An Assessment of Citation Linkages between Patents and Research Papers , 2000, Scientometrics.

[100]  Liwen Vaughan,et al.  Relationship between links to journal Web sites and impact factors , 2002, Aslib Proc..

[101]  Mike Thelwall,et al.  Substance without citation: evaluating the online impact of grey literature , 2014, Scientometrics.

[102]  C. Neylon,et al.  Article-Level Metrics and the Evolution of Scientific Impact , 2009, PLoS biology.

[103]  Cameron Barnes,et al.  The Use of Altmetrics as a Tool for Measuring Research Impact , 2015 .

[104]  Ludo Waltman,et al.  F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations , 2014, J. Assoc. Inf. Sci. Technol..

[105]  P. Seglen,et al.  Citation rates and journal impact factors are not suitable for evaluation of research. , 1998, Acta orthopaedica Scandinavica.