Web indicators for research evaluation. Part 3: books and non standard outputs

This literature review describes web indicators for the impact of books, software, datasets, videos and other non-standard academic outputs. Although journal articles dominate academic research in the health and natural sciences, other types of outputs can make equally valuable contributions to scholarship and are more common in other fields. It is not always possible to get useful citation-based impact indicators for these due to their absence from, or incomplete coverage in, traditional citation indexes. In this context, the web is particularly valuable as a potential source of impact indicators for non-standard academic outputs. The main focus in this review is on books because of the much greater amount of relevant research for them and because they are regarded as particularly valuable in the arts and humanities and in some areas of the social sciences.

[1]  Jennifer Thompson,et al.  The Death of the Scholarly Monograph in the Humanities? Citation Patterns in Literary Scholarship , 2002 .

[2]  Debora Shaw,et al.  An Analysis of the Relationship between Book Reviews and Fiction Holdings in OCLC. , 1991 .

[3]  A. J. M. Linmans,et al.  Why with bibliometrics the Humanities does not need to be the weakest link , 2010, Scientometrics.

[4]  Dawid Weiss,et al.  Measuring success of open source projects using web search engines , 2005 .

[5]  Mike Thelwall,et al.  The influence of time and discipline on the magnitude of correlations between citation counts and quality scores , 2015, J. Informetrics.

[6]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 2: Social media metrics , 2015 .

[7]  Rens Bod,et al.  Can we rank scholarly book publishers? A bibliometric experiment with the field of history , 2015, J. Assoc. Inf. Sci. Technol..

[8]  Kevin Crowston,et al.  Defining Open Source Software Project Success , 2003, ICIS.

[9]  Roberto Cornacchia,et al.  Altmetrics for the humanities: Comparing Goodreads reader ratings with citations to history books , 2015, Aslib J. Inf. Manag..

[10]  Kevin Crowston,et al.  Towards a Portfolio of FLOSS Project Success Measures , 2004, ICSE 2004.

[11]  R. Poldrack,et al.  The publication and reproducibility challenges of shared data , 2015, Trends in Cognitive Sciences.

[12]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[13]  Alesia Zuccalá,et al.  Correlating Libcitations and Citations in the Humanities with WorldCat and Scopus Data , 2015, ISSI.

[14]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[15]  Mike Thelwall,et al.  An automatic method for extracting citations from Google Books , 2015, J. Assoc. Inf. Sci. Technol..

[16]  Daniel Torres-Salinas,et al.  Mapping citation patterns of book chapters in the Book Citation Index , 2012, J. Informetrics.

[17]  Mike Thelwall,et al.  The role of online videos in research communication: A content analysis of YouTube videos cited in academic publications , 2012, J. Assoc. Inf. Sci. Technol..

[18]  Juan Gorraiz,et al.  The power of book reviews: a simple and transparent enhancement approach for book citation indexes , 2013, Scientometrics.

[19]  James Hartley,et al.  Reading and writing book reviews across the disciplines , 2006, J. Assoc. Inf. Sci. Technol..

[20]  Nicolás Robinson-García,et al.  Towards a Book Publishers Citation Reports. First approach using the Book Citation Index , 2012, Revista española de Documentación Científica.

[21]  Gerald Benoît,et al.  Link analysis: An information science approach , 2006, J. Assoc. Inf. Sci. Technol..

[22]  Andrew Philip Weiss,et al.  Assessing the coverage of Hawaiian and Pacific books in the Google Books Digitization Project , 2013, OCLC Syst. Serv..

[23]  Christine L. Borgman,et al.  The conundrum of sharing research data , 2012, J. Assoc. Inf. Sci. Technol..

[24]  Jack Meadows,et al.  Editorial Peer Review: Its Strengths and Weaknesses , 2002, J. Documentation.

[25]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[26]  Daniel Torres-Salinas,et al.  Library Catalog Analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in Economics , 2009, J. Informetrics.

[27]  L. Butler,et al.  Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: an economics pilot study , 2007 .

[28]  C. Tenopir,et al.  Data Sharing by Scientists: Practices and Perceptions , 2011, PloS one.

[29]  Mike Thelwall,et al.  Substance without citation: evaluating the online impact of grey literature , 2014, Scientometrics.

[30]  Mike Thelwall,et al.  Measuring the web impact of digitised scholarly resources , 2012, J. Documentation.

[31]  Alesia Zuccalá,et al.  The Evaluation of Scholarly Books as a Research Output. Current Developments in Europe , 2015, ISSI.

[32]  Heather A. Piwowar,et al.  Sharing Detailed Research Data Is Associated with Increased Citation Rate , 2007, PloS one.

[33]  R. Steen.  Retractions in the scientific literature: do authors deliberately commit research fraud? , 2010, Journal of Medical Ethics.

[34]  Paul Metz,et al.  A Reputational Study of Academic Publishers. , 1996 .

[35]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web , 2015 .

[36]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[37]  C. Nyquist,et al.  An Academic Librarian's Response to the “ITHAKA Faculty Survey 2009: Key Strategic Insights for Libraries, Publishers, and Societies” , 2010 .

[38]  Diana Hicks,et al.  The difficulty of achieving full coverage of international social science literature and the bibliometric consequences , 1999, Scientometrics.

[39]  Mike Thelwall,et al.  Alternative Metrics for Book Impact Assessment: Can Choice Reviews be a Useful Source? , 2015, ISSI.

[40]  Mike Thelwall,et al.  Can Amazon.com reviews help to assess the wider impacts of books? , 2016, J. Assoc. Inf. Sci. Technol..

[41]  Peter Ingwersen,et al.  Indicators for the Data Usage Index (DUI): an incentive for publishing primary biodiversity data through global information infrastructure , 2011, BMC Bioinformatics.

[42]  Ryan James,et al.  An Assessment of the Legibility of Google Books , 2010 .

[43]  Andrew Philip Weiss,et al.  An Examination of Massive Digital Libraries' Coverage of Spanish Language Materials: Issues of Multi-lingual Accessibility in a Decentralized, Mass-Digitized World , 2013, 2013 International Conference on Culture and Computing.

[44]  S. Bramble.  Image analysis for the biological sciences , 1996 .

[45]  Mike Thelwall,et al.  Scholars on soap boxes: Science communication and dissemination in TED videos , 2013, J. Assoc. Inf. Sci. Technol..

[46]  David Dagan Feng,et al.  A Web-based Collaborative System for Medical Image Analysis and Diagnosis , 2000, VIP.

[47]  Evaristo Jiménez-Contreras,et al.  Most borrowed is most cited? Library loan statistics as a proxy for monograph selection in citation indexes , 2013, ArXiv.

[48]  James C. Garand,et al.  Ranking Scholarly Publishers in Political Science: An Alternative Approach , 2011, PS: Political Science & Politics.

[49]  Mike Thelwall,et al.  Linked title mentions: a new automated link search candidate , 2014, Scientometrics.

[50]  Xiaotian Chen,et al.  Google Books and WorldCat: a comparison of their content , 2012 .

[51]  Lillian Lee,et al.  Opinion Mining and Sentiment Analysis , 2008, Found. Trends Inf. Retr..

[52]  J. Nicolaisen.  The scholarliness of published peer reviews: a bibliometric study of book reviews in selected social science fields , 2002 .

[53]  Kim Holmberg.  Online Attention of Universities in Finland: Are the Bigger Universities Bigger Online too? , 2015, ISSI.

[54]  E. Giménez-Toledo,et al.  Evaluation of scientific books’ publishers in social sciences and humanities: Results of a survey , 2013 .

[55]  Erjia Yan,et al.  A Bootstrapping Method to Assess Software Impact in Full-Text Papers , 2015, ISSI.

[56]  Liwen Vaughan,et al.  Web data as academic and business quality estimates: A comparison of three data sources , 2012, J. Assoc. Inf. Sci. Technol..

[57]  Mike Thelwall,et al.  Arts and humanities research evaluation: no metrics please, just data , 2015, J. Documentation.

[58]  Mike Thelwall,et al.  Can the impact of non‐Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia , 2014, J. Assoc. Inf. Sci. Technol..

[59]  Mike Thelwall,et al.  Commenting on YouTube videos: From guatemalan rock to El Big Bang , 2012, J. Assoc. Inf. Sci. Technol..

[60]  Wolfgang Glänzel,et al.  Opportunities for and limitations of the Book Citation Index , 2013, J. Assoc. Inf. Sci. Technol..

[61]  Christian Gumpenberger,et al.  Going beyond Citations: SERUM — a new Tool Provided by a Network of Libraries , 2010 .

[62]  Mike Thelwall,et al.  Can the impact of scholarly images be assessed online? An exploratory study using image identification technology , 2010 .

[63]  Alesia A. Zuccala,et al.  Comparing book citations in humanities journals to library holdings : scholarly use versus 'perceived cultural benefit' (RIP) , 2013 .

[64]  Jim Taylor,et al.  Peer assessment of research : how many publications per staff? , 2009 .

[65]  Bradford A. Hawkins,et al.  Willing or unwilling to share primary biodiversity data: results and implications of an international survey , 2012 .

[66]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009 .

[67]  Mike Thelwall,et al.  General patterns of tag usage among university groups in Flickr , 2008, Online Inf. Rev..

[68]  Matthew S. Mayernik,et al.  Peer Review of Datasets: When, Why, and How , 2015 .

[69]  Evaristo Jiménez-Contreras,et al.  The BiPublishers ranking: Main results and methodological problems when constructing rankings of academic publishers , 2015, ArXiv.

[70]  Fletcher T. H. Cole,et al.  Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences , 2009 .

[71]  Giancarlo Succi,et al.  Download Patterns and Releases in Open Source Software Projects: A Perfect Symbiosis? , 2010, OSS.

[72]  Chengzhi Zhang,et al.  Can Book Reviews be Used to Evaluate Books' Influence? , 2015, ISSI.

[73]  Michael D. Cooper,et al.  Using Article Photocopy Data in Bibliographic Models for Journal Collection Management , 1994, The Library Quarterly.

[74]  Evaristo Jiménez-Contreras,et al.  Analyzing data citation practices according to the Data Citation Index , 2015, ArXiv.

[75]  M. Tsay.  The relationship between journal use in a medical library and citation use. , 1998, Bulletin of the Medical Library Association.

[76]  P. Anagnostou,et al.  Research data sharing: Lessons from forensic genetics. , 2013, Forensic science international. Genetics.

[77]  Liwen Vaughan,et al.  Journal of Informetrics , 2022 .

[78]  Wolfgang Glänzel,et al.  Book Bibliometrics - A New Perspective and Challenge in Indicator Building Based on the Book Citation Index , 2015, ISSI.

[79]  Peter Ingwersen,et al.  Towards a data publishing framework for primary biodiversity data: challenges and potentials for the biodiversity informatics community , 2009, BMC Bioinformatics.

[80]  Hailey Mooney,et al.  The Anatomy of a Data Citation: Discovery, Reuse, and Credit , 2012 .

[81]  Liwen Vaughan.  An Alternative Data Source for Web Hyperlink Analysis: “Sites Linking In” at Alexa Internet , 2012 .

[82]  Martyn Poliakoff,et al.  The Periodic Table of Videos , 2011, Science.