Can Amazon.com reviews help to assess the wider impacts of books?

Although citation counts are often used to evaluate the research impact of academic publications, they are problematic for books that aim for educational or cultural impact. To fill this gap, this article assesses whether a number of simple metrics derived from Amazon.com reviews of academic books could provide evidence of their impact. Based on a set of 2,739 academic monographs from 2008 and a set of 1,305 best‐selling books in 15 Amazon.com academic subject categories, the existence of significant but low or moderate correlations between citations and numbers of reviews, combined with other evidence, suggests that online book reviews tend to reflect the wider popularity of a book rather than its academic impact, although there are substantial disciplinary differences. Metrics based on online reviews are therefore recommended for the evaluation of books that aim at a wide audience inside or outside academia when it is important to capture the broader impacts of educational or cultural activities and when they cannot be manipulated in advance of the evaluation.

[1]  Charles Oppenheim,et al.  The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology , 1997, J. Documentation.

[2]  Amanda Spink,et al.  Use of Scholarly Book Reviews: Implications for Electronic Publishing and Scholarly Communication , 1998, J. Am. Soc. Inf. Sci..

[3]  A. Raan,et al.  A bibliometric analysis of six economics research groups: A comparison with peer review , 1993 .

[4]  Karen S Calhoun,et al.  The Changing Nature of the Catalog and its Integration with Other Discovery Tools: Draft 2B , 2006 .

[5]  Tim Brody,et al.  Earlier Web usage statistics as predictors of later citation impact: Research Articles , 2006 .

[6]  James E. Bessen,et al.  Patent Failure: How Judges, Bureaucrats, and Lawyers Put Innovators at Risk , 2008 .

[7]  Elizabeth Aversa,et al.  The web of knowledge: a festschrift in honor of Eugene Garfield , 2001 .

[8]  Anindya Ghose,et al.  Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets , 2008, Inf. Syst. Res..

[9]  Karen H. Calhoun,et al.  The changing nature of the catalog and its integration with other discovery tools: final report , 2006 .

[10]  M. Porta,et al.  Book citations: influence of epidemiologic thought in the academic community. , 2006, Revista de saude publica.

[11]  Judit Bar-Ilan,et al.  Beyond citations: Scholars' visibility on the social Web , 2012, ArXiv.

[12]  Anthony F. J. van Raan.  Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups , 2013, Scientometrics.

[13]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[14]  N. Bury,et al.  Metal contamination in aquatic environments: science and lateral management , 2009 .

[15]  Juan Gorraiz,et al.  The power of book reviews: a simple and transparent enhancement approach for book citation indexes , 2013, Scientometrics.

[16]  Charles Oppenheim,et al.  The Correlation between citation counts and the 1992 Research Assessment Exercise Ratings for British Library and Information Science University departments , 1995, J. Documentation.

[17]  Diana Hicks,et al.  The difficulty of achieving full coverage of international social science literature and the bibliometric consequences , 1999, Scientometrics.

[18]  Michael D. Smith,et al.  All Reviews are Not Created Equal: The Disaggregate Impact of Reviews and Reviewers at Amazon.Com , 2008 .

[19]  Juan Miguel Campanario,et al.  Coverage, field specialisation and the impact of scientific publishers indexed in the Book Citation Index , 2013, Online Inf. Rev..

[20]  ThelwallMike,et al.  Sentiment strength detection in short informal text , 2010 .

[21]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[22]  Juris Dilevko,et al.  Investigating the Value of Scholarly Book Reviews for the Work of Academic Reference Librarians , 2006 .

[23]  Maarten van Someren,et al.  A machine‐learning approach to coding book reviews as quality indicators: Toward a theory of megacitation , 2014, J. Assoc. Inf. Sci. Technol..

[24]  Paul Metz,et al.  A Reputational Study of Academic Publishers. , 1996 .

[25]  Kate Wong.  A universe from nothing: Why there is something rather than nothing. , 2012 .

[26]  Julian Warner,et al.  The Role of monographs in scholarly Communication: an Empirical Study of Philosophy, Sociology and Economics , 1996, J. Documentation.

[27]  Wolfgang Glänzel,et al.  A Bibliometric Study of Reference Literature in the Sciences and Social Sciences , 1999, Inf. Process. Manag..

[28]  Carole L. Palmer,et al.  Scholarly work and the shaping of digital access , 2005, J. Assoc. Inf. Sci. Technol..

[29]  Wolfgang Glänzel,et al.  Opportunities for and limitations of the Book Citation Index , 2013, J. Assoc. Inf. Sci. Technol..

[30]  Thed N. van Leeuwen,et al.  Book reviews in humanities research evaluations , 2011, J. Assoc. Inf. Sci. Technol..

[31]  Debora Shaw,et al.  An Analysis of the Relationship between Book Reviews and Fiction Holdings in OCLC. , 1991 .

[32]  Loet Leydesdorff,et al.  Edited Volumes, Monographs and Book Chapters in the Book Citation Index (BKCI) and Science Citation Index (SCI, Sosci, A&HCI) , 2012, J. Sci. Res..

[33]  Amanda Spink,et al.  Use of Scholarly Book Reviews: Implications for Electronic Publishing and Scholarly Communication , 1998, J. Am. Soc. Inf. Sci..

[34]  Stevan Harnad,et al.  Earlier Web Usage Statistics as Predictors of Later Citation Impact , 2005, J. Assoc. Inf. Sci. Technol..

[35]  Jackie Rees Ulmer,et al.  Competing for Attention: An Empirical Study of Online Reviewers' Strategic Behavior , 2015, MIS Q..

[36]  Vincent Larivière,et al.  The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities , 2006, J. Assoc. Inf. Sci. Technol..

[37]  Blaise Cronin,et al.  Comparative citation rankings of authors in monographic and journal literature: a study of sociology , 1997, J. Documentation.

[38]  J. Hartley.  Reading and writing book reviews across the disciplines: Research Articles , 2006 .

[39]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[40]  Vincent Larivière,et al.  The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities: Research Articles , 2006 .

[41]  Henk F. Moed,et al.  Library Catalog Analysis is a Useful Tool in Studies of Social Sciences and Humanities , 2008 .

[42]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[43]  Rens Bod,et al.  Can we rank scholarly book publishers? A bibliometric experiment with the field of history , 2015, J. Assoc. Inf. Sci. Technol..

[44]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[45]  Rodrigo Costas,et al.  How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications , 2014, Scientometrics.

[46]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[47]  Antti Ripatti.  MONETARY POLICY, INFLATION AND THE BUSINESS CYCLE , 2010 .

[48]  Kari-Jouko Räihä,et al.  Case amazon: ratings and reviews as part of recommendations , 2007, RecSys '07.

[49]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[50]  Jennifer Thompson,et al.  The Death of the Scholarly Monograph in the Humanities? Citation Patterns in Literary Scholarship , 2002 .

[51]  Günter Krampen,et al.  On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications , 2007, Scientometrics.

[52]  Reginald L. Bell,et al.  Leaders as Catalysts for Organizational Change: How Popular Press Business Books Address this Topic , 2012 .

[53]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[54]  Jeanne M Brown,et al.  Standards for Libraries in Higher Education , 2011 .

[55]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[56]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008, J. Assoc. Inf. Sci. Technol..

[57]  Nicolás Robinson-García,et al.  Towards a Book Publishers Citation Reports. First approach using the Book Citation Index , 2012, Revista española de Documentación Científica.

[58]  L PalmerCarole.  Scholarly work and the shaping of digital access , 2005 .

[59]  Yu-Wei Chang,et al.  Characteristics of research output in social sciences and humanities: From a research evaluation perspective , 2008, J. Assoc. Inf. Sci. Technol..

[60]  Rong Zheng,et al.  The impact of word-of-mouth on book sales: review, blog or tweet? , 2012, ICEC '12.

[61]  A. J. M. Linmans,et al.  Why with bibliometrics the Humanities does not need to be the weakest link , 2010, Scientometrics.

[62]  M. Keary.  The Web of Knowledge: A Festschrift in Honor of Eugene Garfield , 2001 .

[63]  Marc Sageman,et al.  Leaderless Jihad , 2011 .

[64]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[65]  Jeppe Nicolaisen,et al.  The J-shaped distribution of citedness , 2002, J. Documentation.

[66]  ThelwallMike,et al.  Sentiment strength detection for the social web , 2012 .

[67]  B. Cronin,et al.  The web of knowledge: a festschrift in honor of Eugene Garfield , 2000 .

[68]  D. Aksnes,et al.  Peer reviews and bibliometric indicators: a comparative study at a Norwegian university , 2004 .

[69]  Linda Butler,et al.  Extending citation analysis to non-source items , 2006, Scientometrics.

[70]  Daniel Torres-Salinas,et al.  Library Catalog Analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in Economics , 2009, J. Informetrics.

[71]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[72]  Jean Tague-Sutcliffe,et al.  An Introduction to Informetrics , 1992, Inf. Process. Manag..

[73]  Massimo Franceschet,et al.  The first Italian research assessment exercise: A bibliometric perspective , 2009, J. Informetrics.

[74]  L. Butler,et al.  Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: an economics pilot study , 2007 .

[75]  Greg Myers,et al.  Lexical cohesion and specialized knowledge in science and popular science texts , 1991 .

[76]  Jim Taylor,et al.  Peer assessment of research : how many publications per staff? , 2009 .

[77]  Fletcher T. H. Cole,et al.  Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences , 2009, J. Assoc. Inf. Sci. Technol..

[78]  Evaristo Jiménez-Contreras,et al.  Most borrowed is most cited? Library loan statistics as a proxy for monograph selection in citation indexes , 2013, ArXiv.

[79]  E. Giménez-Toledo,et al.  Evaluation of scientific books’ publishers in social sciences and humanities: Results of a survey , 2013 .

[80]  Charles Oppenheim,et al.  The Influence of Peer Review on the Research Assessment Exercise , 2004, J. Inf. Sci..

[81]  Tim C. E. Engels,et al.  A label for peer-reviewed books , 2013, J. Assoc. Inf. Sci. Technol..

[82]  J. Barendregt,et al.  Global burden of disease , 1997, The Lancet.

[83]  Yu-Wei Chang,et al.  Characteristics of research output in social sciences and humanities: From a research evaluation perspective , 2008 .

[84]  Ves Thomas.  THE POWER OF THE BOOK. , 1967 .

[85]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009 .

[86]  James C. Garand,et al.  Ranking Scholarly Publishers in Political Science: An Alternative Approach , 2011, PS: Political Science & Politics.

[87]  Mike Thelwall,et al.  Sentiment strength detection for the social web , 2012, J. Assoc. Inf. Sci. Technol..

[88]  Elea Giménez-Toledo,et al.  Peer review and in-depth interviews with publishers as a means of assessing quality of research monographs , 2008 .

[89]  Julian Warner,et al.  A critical review of the application of citation studies to the Research Assessment Exercises , 2000, J. Inf. Sci..

[90]  John Cullars,et al.  Citation characteristics of English-language monographs in philosophy , 1998 .

[91]  Charles Oppenheim,et al.  Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE , 2003, J. Documentation.

[92]  Rodrigo Costas,et al.  Users, narcissism and control – tracking the impact of scholarly publications in the 21st century , 2012 .

[93]  Judit Bar-Ilan,et al.  Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar , 2010, Scientometrics.

[94]  D. Hicks.  Performance-based university research funding systems , 2012 .

[95]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[96]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1992, Scientometrics.

[97]  Daryl E. Chubin,et al.  Content Analysis of References: Adjunct or Alternative to Citation Counting? , 1975 .

[98]  Michael H. MacRoberts,et al.  Problems of citation analysis: A study of uncited and seldom-cited influences , 2010, J. Assoc. Inf. Sci. Technol..

[99]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[100]  Carole L. Palmer,et al.  Scholarly work and the shaping of digital access: Research Articles , 2005 .

[101]  Nikolaos Korfiatis,et al.  The Influences of Negativity and Review Quality on the Helpfulness of Online Reviews , 2011, ICIS.

[102]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[103]  James Hartley,et al.  Reading and writing book reviews across the disciplines , 2006, J. Assoc. Inf. Sci. Technol..

[104]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[105]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[106]  Peter Williams,et al.  The role and future of the monograph in arts and humanities research , 2009, Aslib Proc..

[107]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009, J. Assoc. Inf. Sci. Technol..

[108]  Mike Thelwall,et al.  An automatic method for extracting citations from Google Books , 2015, J. Assoc. Inf. Sci. Technol..

[109]  Alesia A. Zuccala,et al.  Comparing book citations in humanities journals to library holdings : scholarly use versus 'perceived cultural benefit' (RIP) , 2013 .

[110]  Eric H. J. Spruyt,et al.  Changing publication patterns in the Social Sciences and Humanities, 2000–2009 , 2012, Scientometrics.

[111]  Daniel Torres-Salinas,et al.  Mapping citation patterns of book chapters in the Book Citation Index , 2012, J. Informetrics.

[112]  Lynn Silipigni Connaway,et al.  Estimating the audience level for library resources , 2008, J. Assoc. Inf. Sci. Technol..

[113]  Johan Bollen,et al.  Toward alternative metrics of journal impact: A comparison of download and citation data , 2005, Inf. Process. Manag..

[114]  唐戸 俊一郎,et al.  Deformation of Earth Materials : an Introduction to the Rheology of Solid Earth , 2008 .