Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web

The extensive use of the web by many sectors of society has created the potential for new wider impact indicators. This article reviews research about Google Scholar and Google Patents, both of which can be used as sources of impact indicators for academic articles. It also briefly reviews methods to extract types of links and citations from the web as a whole, although the indicators that these generate are now probably too broad and too dominated by automatically generated websites, such as library and publisher catalogues, to be useful in practice. More valuable web-based indicators can be derived from specific types of web pages that cite academic research, such as online presentations, course syllabi, and science blogs. These provide evidence that is easier to understand and use and less likely to be affected by unwanted types of automatically generated content, although they are susceptible to gaming.

[1]  James Wilsdon The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management , 2016 .

[2]  Diana Hicks,et al.  The difficulty of achieving full coverage of international social science literature and the bibliometric consequences , 1999, Scientometrics.

[3]  Gustavo Lannelongue,et al.  Scholarly Impact Revisited , 2012 .

[4]  Péter Jacsó,et al.  The pros and cons of computing the h-index using Scopus , 2008, Online Inf. Rev..

[5]  Mike Thelwall,et al.  Scholarly Use of the Web: What Are the Key Inducers of Links to Journal Web Sites , 2003, J. Assoc. Inf. Sci. Technol..

[6]  Mike Thelwall,et al.  Substance without citation: evaluating the online impact of grey literature , 2014, Scientometrics.

[7]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1992, Scientometrics.

[8]  R. Kling,et al.  Scholarly communication and the continuum of electronic publishing , 1999 .

[9]  Jaime A. Teixeira da Silva,et al.  The need for post-publication peer review in plant science publishing , 2013, Front. Plant Sci..

[10]  Peder Olesen Larsen,et al.  The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index , 2010, Scientometrics.

[11]  Monika Henzinger,et al.  The stability of the h-index , 2009, Scientometrics.

[12]  R. Whitley The Intellectual and Social Organization of the Sciences (Second Edition: with new introductory chapter entitled 'Science Transformed? The Changing Nature of Knowledge Production at the End of the Twentieth Century') , 1984 .

[13]  Mike Thelwall,et al.  Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus , 2011, J. Assoc. Inf. Sci. Technol..

[14]  Michael H. MacRoberts,et al.  Problems of citation analysis , 1996, Scientometrics.

[15]  Nisa Bakkalbasi,et al.  An Examination of Citation Counts in a New Scholarly Communication Environment , 2005, D Lib Mag..

[16]  Vincent Larivière,et al.  Conference proceedings as a source of scientific information: A bibliometric analysis , 2008, J. Assoc. Inf. Sci. Technol..

[17]  Mike Thelwall,et al.  Motivations for URL citations to open access library and information science articles , 2006, Scientometrics.

[18]  Peter Ingwersen,et al.  The calculation of web impact factors , 1998, J. Documentation.

[19]  Tobias Siebenlist,et al.  Applying social bookmarking data to evaluate journal usage , 2011, J. Informetrics.

[20]  José Luis Ortega,et al.  Science is all in the eye of the beholder: Keyword maps in Google scholar citations , 2012, J. Assoc. Inf. Sci. Technol..

[21]  Yvonne Rogers,et al.  Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science , 2008, J. Assoc. Inf. Sci. Technol..

[22]  Judit Bar-Ilan,et al.  Twelve years of Wikipedia research , 2014, WebSci '14.

[23]  S. D. De Groote,et al.  Coverage of Google Scholar, Scopus, and Web of Science: a case study of the h-index in nursing. , 2012, Nursing outlook.

[24]  Narongrit Sombatsompop,et al.  Making an equality of ISI impact factors for different subject fields , 2005, J. Assoc. Inf. Sci. Technol..

[25]  Vincent Larivière,et al.  Benchmarking scientific output in the social sciences and humanities: The limits of existing databases , 2006, Scientometrics.

[26]  Enrique Orduña-Malea,et al.  Google Scholar Metrics evolution: an analysis according to languages , 2013, Scientometrics.

[27]  Henk F. Moed,et al.  Statistical relationships between downloads and citations at the level of individual documents within a single journal: Book Reviews , 2005 .

[28]  Martin Meyer,et al.  What is Special about Patent Citations? Differences between Scientific and Patent Citations , 2000, Scientometrics.

[29]  Stéfan Jacques Darmoni,et al.  Reading factor as a credible alternative to impact factor: a preliminary study , 2000 .

[30]  John Mingers,et al.  Counting the citations: a comparison of Web of Science and Google Scholar in the field of business and management , 2010, Scientometrics.

[31]  Mike Thelwall,et al.  Evaluating altmetrics , 2013, Scientometrics.

[32]  Joshua A Hirsch,et al.  Guidelines warfare over interventional techniques: is there a lack of discourse or straw man? , 2012, Pain physician.

[33]  Mike Thelwall,et al.  Web Impact Factors for Australasian universities , 2002, Scientometrics.

[34]  José Luis Ortega,et al.  Scientific research activity and communication measured with cybermetrics indicators , 2006, J. Assoc. Inf. Sci. Technol..

[35]  José Luis Ortega,et al.  Microsoft academic search and Google scholar citations: Comparative analysis of author profiles , 2014, J. Assoc. Inf. Sci. Technol..

[36]  Martin Meyer,et al.  Patent Citations in a Novel Field of Technology — What Can They Tell about Interactions between Emerging Communities of Science and Technology? , 2000, Scientometrics.

[37]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[38]  Regan A. R. Gurung,et al.  Predicting Textbook Reading , 2011 .

[39]  Massimo Franceschet,et al.  A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar , 2010, Scientometrics.

[40]  Madian Khabsa,et al.  Digital commons , 2020, Internet Policy Rev..

[41]  Andreas Thor,et al.  Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts , 2009, J. Informetrics.

[42]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[43]  Adam Eyre-Walker,et al.  The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations , 2013, PLoS biology.

[44]  Mike Thelwall,et al.  Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics , 2014, J. Assoc. Inf. Sci. Technol..

[45]  A. Kulkarni,et al.  Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. , 2009, JAMA.

[46]  Mike Thelwall,et al.  The influence of time and discipline on the magnitude of correlations between citation counts and quality scores , 2015, J. Informetrics.

[47]  M. Tsay The relationship between journal use in a medical library and citation use. , 1998, Bulletin of the Medical Library Association.

[48]  Charles Oppenheim,et al.  Comparing alternatives to the Web of Science for coverage of the social sciences' literature , 2007, J. Informetrics.

[49]  C. Nyquist,et al.  An Academic Librarian's Response to the “ITHAKA Faculty Survey 2009: Key Strategic Insights for Libraries, Publishers, and Societies” , 2010 .

[50]  Robert D. Shelton,et al.  Causal Connections between Scientometric Indicators: Which Ones Best Explain High-Technology Manufacturing Outputs? , 2015, ISSI.

[51]  Alfred E. Hartemink,et al.  Citations and the h index of soil researchers and journals in the Web of Science, Scopus, and Google Scholar , 2013, PeerJ.

[52]  Yu-Wei Chang,et al.  Characteristics of research output in social sciences and humanities: From a research evaluation perspective , 2008, J. Assoc. Inf. Sci. Technol..

[53]  Mike Thelwall Journal impact evaluation: a webometric perspective , 2012, Scientometrics.

[54]  Lokman I. Meho,et al.  Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar , 2007, J. Assoc. Inf. Sci. Technol..

[55]  Mike Thelwall,et al.  Using the Web for research evaluation: The Integrated Online Impact indicator , 2010, J. Informetrics.

[56]  Matthew E Falagas,et al.  Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses , 2007, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[57]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[58]  Blaise Cronin,et al.  Bibliometrics and beyond: some thoughts on web-based citation analysis , 2001, J. Inf. Sci..

[59]  Gail Herrera,et al.  Google Scholar Users and User Behaviors: An Exploratory Study , 2011, Coll. Res. Libr..

[60]  Torill Mortensen,et al.  Blogging thoughts: personal publication as an online research tool , 2002 .

[61]  Anne-Wil Harzing,et al.  A longitudinal study of Google Scholar coverage between 2012 and 2013 , 2013, Scientometrics.

[62]  Jack Meadows,et al.  Editorial Peer Review: Its Strengths and Weaknesses , 2002, J. Documentation.

[63]  Debora Shaw,et al.  A new look at evidence of scholarly citation in citation indexes and from web sources , 2008, Scientometrics.

[64]  Cassidy R. Sugimoto,et al.  Web Impact Metrics for Research Assessment , 2014 .

[65]  Jim Taylor,et al.  Peer assessment of research : how many publications per staff? , 2009 .

[66]  Martin Meyer,et al.  Tracing knowledge flows in innovation systems , 2002, Scientometrics.

[67]  Michael H. MacRoberts,et al.  Problems of citation analysis: A critical review , 1989, JASIS.

[68]  Elaine Lasda Bergman,et al.  Finding Citations to Social Work Literature: The Relative Benefits of Using Web of Science, Scopus, or Google Scholar , 2012 .

[69]  Julian Warner,et al.  A critical review of the application of citation studies to the Research Assessment Exercises , 2000, J. Inf. Sci..

[70]  Jaime A. Teixeira da Silva,et al.  The need for post-publication peer review in plant science publishing , 2013, Front. Plant Sci..

[71]  Joost C. F. de Winter,et al.  The expansion of Google Scholar versus Web of Science: a longitudinal study , 2013, Scientometrics.

[72]  Jennifer Mishra,et al.  A Content Analysis of Introductory Courses in Music Education at NASM-Accredited Colleges and Universities , 2011 .

[73]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[74]  Philipp Mayr,et al.  An exploratory study of Google Scholar , 2007, Online Inf. Rev..

[75]  Liz Allen,et al.  Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation , 2012, BMJ Open.

[76]  Nicolás Robinson-García,et al.  The Google scholar experiment: How to index false papers and manipulate bibliometric indicators , 2013, J. Assoc. Inf. Sci. Technol..

[77]  Mike Thelwall,et al.  Linked title mentions: a new automated link search candidate , 2014, Scientometrics.

[78]  Massimo Franceschet,et al.  The first Italian research assessment exercise: A bibliometric perspective , 2009, J. Informetrics.

[79]  Mike Thelwall,et al.  Guideline references and academic citations as evidence of the clinical value of health research , 2016, J. Assoc. Inf. Sci. Technol..

[80]  Enrique Orduña-Malea,et al.  Methods for estimating the size of Google Scholar , 2014, Scientometrics.

[81]  Martin S. Meyer,et al.  Patent citation analysis in a novel field of technology:An exploration of nano-science and nano-technology , 2001, Scientometrics.

[82]  Gordana Brumini,et al.  Examining the Medical Blogosphere: An Online Survey of Medical Bloggers , 2008, Journal of medical Internet research.

[83]  Anne-Wil Harzing,et al.  A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners , 2013, Scientometrics.

[84]  Tony Stankus,et al.  Handle With Care: Use and Citation Data for Science Journal Management , 1982, Scientific Journals: Issues in Library Selection and Management.

[85]  S. A. Evans,et al.  Multicultural Competence and Social Justice Training in Counseling Psychology and Counselor Education , 2009 .

[86]  Nabil Amara,et al.  Counting citations in the field of business and management: why use Google Scholar rather than the Web of Science , 2012, Scientometrics.

[87]  Alastair Smith,et al.  A Tale of Two Web Spaces: Comparing Sites Using Web Impact Factors. , 1999 .

[88]  Johan Bollen,et al.  Usage bibliometrics , 2011, Annu. Rev. Inf. Sci. Technol..

[89]  María José Luzón,et al.  Scholarly hyperwriting: The function of links in academic weblogs , 2009, J. Assoc. Inf. Sci. Technol..

[90]  David B. Daniel,et al.  Textbook Use and Learning: A North American Perspective , 2012 .

[91]  Liwen Vaughan,et al.  Relationship between links to journal Web sites and impact factors , 2002, Aslib Proc..

[92]  Stephen S. Murray,et al.  The bibliometric properties of article readership information , 2005, J. Assoc. Inf. Sci. Technol..

[93]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008 .

[94]  Juan Gorraiz,et al.  Comparison of citation and usage indicators: the case of oncology journals , 2010, Scientometrics.

[95]  Mike Thelwall,et al.  The connection between the research of a university and counts of links to its web pages: An investigation based upon a classification of the relationships of pages to the research of the host university , 2003, J. Assoc. Inf. Sci. Technol..

[96]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[97]  Blaise Cronin,et al.  Invoked on the Web , 1998, J. Am. Soc. Inf. Sci..

[98]  R. Rousseau Sitations: an exploratory study , 1997 .

[99]  Susanne Mikki,et al.  Comparing Google Scholar and ISI Web of Science for Earth Sciences , 2010, Scientometrics.

[100]  Stephen P. Harter,et al.  Web-based analyses of E-journal impact: Approaches, problems, and issues , 2000, J. Am. Soc. Inf. Sci..

[101]  Mike Thelwall,et al.  An automatic method for assessing the teaching impact of books from online academic syllabi , 2016, J. Assoc. Inf. Sci. Technol..

[102]  Péter Jacsó,et al.  Metadata mega mess in Google Scholar , 2010, Online Inf. Rev..

[103]  Dominique Brossard,et al.  Science News Consumption Patterns and Their Implications for Public Understanding of Science , 2015 .

[104]  Cyril Labbé,et al.  Duplicate and fake publications in the scientific literature: how many SCIgen papers in computer science? , 2012, Scientometrics.

[105]  Paul Groth,et al.  Studying Scientific Discourse on the Web Using Bibliometrics: A Chemistry Blogging Case Study , 2010 .

[106]  Henk F. Moed,et al.  Statistical relationships between downloads and citations at the level of individual documents within a single journal , 2005, J. Assoc. Inf. Sci. Technol..

[107]  Ian Rowlands,et al.  The missing link: journal usage metrics , 2007, Aslib Proc..

[108]  Wolfgang Glänzel,et al.  A bibliometric study on ageing and reception processes of scientific literature , 1995, J. Inf. Sci..

[109]  Finn Årup Nielsen,et al.  Scientific citations in Wikipedia , 2007, First Monday.

[110]  A. Neely,et al.  Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001 , 2008 .

[111]  Mike Thelwall,et al.  Extracting macroscopic information from Web links , 2001, J. Assoc. Inf. Sci. Technol..

[112]  Péter Jacsó,et al.  Google Scholar duped and deduped – the aura of “robometrics” , 2011 .

[113]  G. Kirkup Academic blogging: academic practice and academic identity , 2010 .

[114]  Péter Jacsó,et al.  Google Scholar revisited , 2008, Online Inf. Rev..

[115]  I. Mewburn,et al.  Why do academics blog? An analysis of audiences, purposes and challenges , 2013 .

[116]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[117]  María José Luzón,et al.  Academic Weblogs as Tools for E-Collaboration Among Researchers , 2009 .

[118]  Tony Stankus,et al.  Wikipedia, Scholarpedia, and References to Journals in the Brain and Behavioral Sciences: A Comparison of Cited Sources and Recommended Readings in Matching Free Online Encyclopedia Entries , 2010 .

[119]  John D. McDonald,et al.  Understanding journal usage: A statistical analysis of citation and use , 2007, J. Assoc. Inf. Sci. Technol..

[120]  Rob Kling,et al.  Not just a matter of time: Field differences and the shaping of electronic media in supporting scientific communication , 1999, J. Am. Soc. Inf. Sci..

[121]  Christian Gumpenberger,et al.  Going beyond Citations: SERUM — a new Tool Provided by a Network of Libraries , 2010 .

[122]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[123]  Ronald Rousseau,et al.  The journal download immediacy index (DII): experiences using a Chinese full-text database , 2010, Scientometrics.

[124]  Wolfgang Glänzel,et al.  Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon , 2004, Scientometrics.

[125]  Johan Bollen,et al.  Towards usage-based impact metrics: first results from the mesur project. , 2008, JCDL '08.

[126]  Rory Ewins Who are You? Weblogs and Academic Identity , 2005 .

[127]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[128]  Mike Thelwall,et al.  Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals , 2008 .

[129]  Bradley M. Hemminger,et al.  Altmetrics in the wild: Using social media to explore scholarly impact , 2012, ArXiv.

[130]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[131]  Mike Thelwall,et al.  How is science cited on the Web? A classification of google unique Web citations , 2007, J. Assoc. Inf. Sci. Technol..

[132]  Péter Jacsó,et al.  Deflated, inflated and phantom citation counts , 2006, Online Inf. Rev..

[133]  David King,et al.  Identifying Audiences of E-Infrastructures - Tools for Measuring Impact , 2012, PloS one.

[134]  Anne-Wil Harzing,et al.  Google Scholar as a new source for citation analysis , 2008 .

[135]  Sara Kjellberg,et al.  I am a blogging researcher: Motivations for blogging in a scholarly context , 2010, First Monday.

[136]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[137]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[138]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[139]  Christy Caldwell,et al.  Shifting Sands: Science Researchers on Google Scholar, Web of Science, and PubMed, with Implications for Library Collections Budgets. , 2010 .

[140]  Mike Thelwall,et al.  Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines , 2008, Scientometrics.

[141]  Judit Bar-Ilan,et al.  Web of Science with the Conference Proceedings Citation Indexes: the case of computer science , 2010, Scientometrics.

[142]  Judit Bar-Ilan,et al.  Which h-index? — A comparison of WoS, Scopus and Google Scholar , 2008, Scientometrics.

[143]  Antal van den Bosch,et al.  A Longitudinal Analysis of Search Engine Index Size , 2015, ISSI.

[144]  Stephen P. Harter,et al.  Web-based analyses of E-journal impact: Approaches, problems, and issues , 2000, J. Am. Soc. Inf. Sci..

[145]  Liwen Vaughan,et al.  Can electronic journal usage data replace citation data as a measure of journal use? An empirical examination , 2006 .