Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research

This study compares Spanish and UK research in eight subject fields using a range of bibliometric and social media indicators. For each field, lists of Spanish and UK journal articles published in the year 2012 and their citation counts were extracted from Scopus. The software Webometric Analyst was then used to extract a range of altmetrics for these articles, including patent citations, online presentation mentions, online course syllabus mentions, Wikipedia mentions and Mendeley reader counts and Altmetric.com was used to extract Twitter mentions. Results show that Mendeley is the altmetric source with the highest coverage, with 80 % of sampled articles having one or more Mendeley readers, followed by Twitter (34 %). The coverage of the remaining sources was lower than 3 %. All of the indicators checked either have too little data or increase the overall difference between Spain and the UK and so none can be suggested as alternatives to reduce the bias against Spain in traditional citation indexes.

[1]  Martin Meyer,et al.  What is Special about Patent Citations? Differences between Scientific and Patent Citations , 2000, Scientometrics.

[2]  Rodrigo Costas,et al.  Assessing the Impact of Publications Saved by Mendeley Users: Is There Any Different Pattern Among Users? , 2014 .

[3]  J. Martínez,et al.  Enciclopedia del español en el mundo: anuario del Instituto Cervantes 2006-2007 , 2006 .

[4]  Mike Thelwall,et al.  An automatic method for assessing the teaching impact of books from online academic syllabi , 2016, J. Assoc. Inf. Sci. Technol..

[5]  Björn Hammarfelt,et al.  Using altmetrics for assessing research impact in the humanities , 2014, Scientometrics.

[6]  Lance B. Kurke,et al.  Designing a Great Syllabus , 1994 .

[7]  Benjamín Vargas-Quesada,et al.  Indicadores Bibliométricos de la Actividad Científica Española 2007. Informe 2009 , 2009 .

[8]  Mike Thelwall,et al.  Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses , 2008 .

[9]  Jacques Michel,et al.  Patent citation analysis.A closer look at the basic input data from patent search reports , 2001, Scientometrics.

[10]  Jesús Rey-Rocha,et al.  Why do I publish research articles in English instead of my own language? Differences in Spanish researchers’ motivations across scientific domains , 2015, Scientometrics.

[11]  Michael Thelwall,et al.  Web Indicators for Research Evaluation: A Practical Guide , 2016, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[12]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 3: books and non standard outputs , 2015 .

[13]  Thed N. van Leeuwen,et al.  Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance , 2001, Scientometrics.

[14]  Mike Thelwall,et al.  Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows , 2014, J. Assoc. Inf. Sci. Technol..

[15]  Vincent Larivière,et al.  Benchmarking scientific output in the social sciences and humanities: The limits of existing databases , 2006, Scientometrics.

[16]  Ronald Rousseau,et al.  A Multi-metric Approach for Research Evaluations , 2022 .

[17]  Lucy Holman Rector Comparison of Wikipedia and other encyclopedias for accuracy, breadth, and depth in historical articles , 2008 .

[18]  Mike Thelwall,et al.  How is science cited on the Web? A classification of google unique Web citations , 2007, J. Assoc. Inf. Sci. Technol..

[19]  Mike Thelwall,et al.  Mendeley readership counts: An investigation of temporal and disciplinary differences , 2016, J. Assoc. Inf. Sci. Technol..

[20]  Mike Thelwall,et al.  Validating online reference managers for scholarly impact measurement , 2011, Scientometrics.

[21]  Mike Thelwall,et al.  Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals , 2008, J. Assoc. Inf. Sci. Technol..

[22]  Victor Henning,et al.  Mendeley - A Last.fm For Research? , 2008, 2008 IEEE Fourth International Conference on eScience.

[23]  Rodrigo Costas,et al.  The thematic orientation of publications mentioned on social media: Large-scale disciplinary comparison of social media metrics with citations , 2015, Aslib J. Inf. Manag..

[24]  Mike Thelwall,et al.  Disciplinary differences in Twitter scholarly communication , 2014, Scientometrics.

[25]  Mike Thelwall,et al.  More precise methods for national research citation impact comparisons , 2015, J. Informetrics.

[26]  Vicente P. Guerrero-Bote,et al.  Journal maps on the basis of Scopus data: A comparison with the Journal Citation Reports of the ISI , 2010 .

[27]  Mike Thelwall,et al.  National research impact indicators from Mendeley readers , 2015, J. Informetrics.

[28]  Peter Kerkhof,et al.  Why Do Scholars Use PowerPoint the Way They Do? , 2015 .

[29]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[30]  Philip M. Davis,et al.  The effect of the Web on undergraduate citation behavior: A 2000 update , 2002 .

[31]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web , 2015 .

[32]  Jennifer Phillips,et al.  Analysis of reference sources used in drug-related Wikipedia articles. , 2015, Journal of the Medical Library Association : JMLA.

[33]  Mike Thelwall,et al.  Web indicators for research evaluation. Part 2: Social media metrics , 2015 .

[34]  Mike Thelwall,et al.  Webometric research with the Bing Search API 2.0 , 2012, J. Informetrics.

[35]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[36]  Curtis J. Bonk,et al.  ONLINE TEACHING IN AN ONLINE WORLD , 2001 .

[37]  Henk F. Moed,et al.  Developing Bibliometric Indicators of Research Performance in Computer Science , 2007 .

[38]  Vincent Larivière,et al.  Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature , 2013, J. Assoc. Inf. Sci. Technol..

[39]  Gunther Eysenbach,et al.  Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact , 2011, Journal of medical Internet research.

[40]  Jason Priem,et al.  How and why scholars cite on Twitter , 2010, ASIST.

[41]  Ali Ghazinejad,et al.  Who Tweets about Science? , 2015, ISSI.

[42]  Blaise Cronin,et al.  Bibliometrics and beyond: some thoughts on web-based citation analysis , 2001, J. Inf. Sci..

[43]  Debora Shaw,et al.  Web citation data for impact assessment: A comparison of four science disciplines , 2005, J. Assoc. Inf. Sci. Technol..

[44]  Johan Bollen,et al.  Toward alternative metrics of journal impact: A comparison of download and citation data , 2005, Inf. Process. Manag..

[45]  Vincent Larivière,et al.  Who reads research articles? An altmetrics analysis of Mendeley user categories , 2015, J. Assoc. Inf. Sci. Technol..

[46]  Enrique Orduña Malea La web social como nuevo medio de comunicación y evaluación científica [Reseña] , 2016 .

[47]  John H. Hummel,et al.  Constructing the Course Syllabus: Faculty and Student Perceptions of Important Syllabus Components. , 1999 .

[48]  David Stuart Social Media Metrics , 2009 .

[49]  Debora Shaw,et al.  Can web citations be a measure of impact? An investigation of journals in the life sciences , 2005, ASIST.

[50]  Thed N. van Leeuwen,et al.  Technological Relevance of Science: An Assessment of Citation Linkages between Patents and Research Papers , 2000, Scientometrics.

[51]  Feza Orhan,et al.  Prospective teachers’ opinions on the value of PowerPoint presentations in lecturing , 2010 .

[52]  George Veletsianos,et al.  Higher education scholars' participation and practices on Twitter , 2012, J. Comput. Assist. Learn..

[53]  D. Nicholas,et al.  Scholarly reputation in the digital age and the role of emerging platforms and mechanisms , 2016 .

[54]  Curtis J. Bonk,et al.  Twenty-first century college syllabi: Options for online communication and interactivity , 2002 .

[55]  Katrin Weller,et al.  Citation Analysis in Twitter: Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences , 2011, #MSM.

[56]  Koenraad Debackere,et al.  Traces of Prior Art: An analysis of non-patent references found in patent documents , 2006, Scientometrics.

[57]  Mike Thelwall,et al.  Mendeley readership altmetrics for medical articles: An analysis of 45 fields , 2016, J. Assoc. Inf. Sci. Technol..

[58]  Daqing He,et al.  User participation in an academic social networking service: A survey of open group users on Mendeley , 2014, J. Assoc. Inf. Sci. Technol..

[59]  Szu-chia S. Lo,et al.  Scientific linkage of science research and technology development: a case of genetic engineering research , 2009, Scientometrics.

[60]  Loet Leydesdorff,et al.  International Collaboration in Science: The Global Map and the Network , 2013, ArXiv.

[61]  Katrin Weller,et al.  Social Software in Academia: Three Studies on Users' Acceptance of Web 2.0 Services , 2010 .

[62]  Lutz Bornmann,et al.  Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics , 2014, J. Informetrics.

[63]  Antal van den Bosch,et al.  Estimating search engine index size variability: a 9-year longitudinal study , 2016, Scientometrics.

[64]  Bradley M. Hemminger,et al.  Scientometrics 2.0: New metrics of scholarly impact on the social Web , 2010, First Monday.

[65]  Mike Thelwall,et al.  Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management , 2015 .

[66]  Henk F. Moed,et al.  Multidimensional assessment of scholarly research impact , 2014, J. Assoc. Inf. Sci. Technol..

[67]  Kim Holmberg,et al.  Altmetrics for Information Professionals: Past, Present and Future , 2015 .

[68]  Martin Meyer,et al.  Academic patents as an indicator of useful research? A new approach to measure academic inventiveness , 2003 .

[69]  William Gunn,et al.  Social Signals Reflect Academic Impact: What it Means When a Scholar Adds a Paper to Mendeley , 2013 .

[70]  Ulrich Schmoch,et al.  Tracing the knowledge transfer from science to technology as reflected in patent indicators , 2005, Scientometrics.

[71]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[72]  Mike Thelwall,et al.  Patent citation analysis with Google , 2017, J. Assoc. Inf. Sci. Technol..

[73]  Luis Deltell Escolar,et al.  La Guía Docente: un reto en el nuevo modelo de educación universitaria , 2012 .

[74]  Hsin-Liang Chen,et al.  The perspectives of higher education faculty on Wikipedia , 2010, Electron. Libr..

[75]  S. Haustein,et al.  Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns , 2015, PloS one.

[76]  Brendan Luyt,et al.  Improving Wikipedia's credibility: References and citations in a sample of history articles , 2010, J. Assoc. Inf. Sci. Technol..

[77]  Henk Eijkman Academics and Wikipedia: Reframing Web 2.0+ as a Disruptor of Traditional Academic Power-Knowledge Arrangements. , 2010 .

[78]  Zaida Chinchilla-Rodríguez,et al.  Coverage analysis of Scopus: A journal metric approach , 2007, Scientometrics.

[79]  Mike Thelwall,et al.  Introduction to Webometrics: Quantitative Web Research for the Social Sciences , 2009, Introduction to Webometrics.

[80]  Elisa Alonso-Jiménez Una aproximación a Wikipedia como polisistema cultural , 2015 .

[81]  John G. Breslin,et al.  Understanding how Twitter is used to spread scientific messages , 2010 .

[82]  Jian Wang,et al.  Citation time window choice for research impact evaluation , 2013, Scientometrics.

[83]  Euan A. Adie,et al.  Altmetric: enriching scholarly content with article‐level discussion and metrics , 2013, Learn. Publ..

[84]  Patricia Jaramillo,et al.  Qué hacer con la tecnología en el aula: inventario de usos de las TIC para aprender y enseñar , 2009 .

[85]  Mike Thelwall,et al.  Search engine coverage bias: evidence and possible causes , 2004, Inf. Process. Manag..

[86]  Nicolás Robinson-García,et al.  New data, new possibilities: Exploring the insides of Altmetric.com , 2014, ArXiv.

[87]  Frans Albarillo,et al.  Language in Social Science Databases: English Versus Non-English Articles in JSTOR and Scopus , 2014 .

[88]  Herwig Rehatschek,et al.  Wikipedia – challenges and new horizons in enhancing medical education , 2015, BMC Medical Education.

[89]  Laura María,et al.  Web 2.0: Wikipedia como fuente de información sobre las ciencias de la alimentación y de la nutrición , 2013 .

[90]  Hyla H. Polen,et al.  Scope, Completeness, and Accuracy of Drug Information in Wikipedia , 2008, The Annals of pharmacotherapy.

[91]  Susan Breakenridge Fink The Many Purposes of Course Syllabi: Which are Essential and Useful? , 2012 .

[92]  Timothy W. Finin,et al.  Why we twitter: understanding microblogging usage and communities , 2007, WebKDD/SNA-KDD '07.

[93]  Eric H. J. Spruyt,et al.  Changing publication patterns in the Social Sciences and Humanities, 2000–2009 , 2012, Scientometrics.

[94]  Rodrigo Costas,et al.  Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective , 2014, J. Assoc. Inf. Sci. Technol..

[95]  Mike Thelwall,et al.  Are wikipedia citations important evidence of the impact of scholarly articles and books? , 2017, J. Assoc. Inf. Sci. Technol..

[96]  Liwen Vaughan,et al.  Web citation data for impact assessment: A comparison of four science disciplines: Book Reviews , 2005 .

[97]  Debora Shaw,et al.  Bibliographic and Web citations: What is the difference? , 2003, J. Assoc. Inf. Sci. Technol..

[98]  Julià Minguillón,et al.  Wikipedia at university: what faculty think and do about it , 2015, Electron. Libr..

[99]  Robert Costanza,et al.  Perceived credibility of Internet encyclopedias , 2011, Comput. Educ..

[100]  Carmen Marta-Lazo,et al.  LA EDUCACIÓN MEDIÁTICA EN LAS TITULACIONES DE EDUCACIÓN Y COMUNICACIÓN DE LAS UNIVERSIDADES ESPAÑOLAS. ANÁLISIS DE LOS RECURSOS RECOMENDADOS EN LAS GUÍAS DOCENTES , 2014 .

[101]  Koenraad Debackere,et al.  Linking science to technology: Using bibliographic references in patents to build linkage schemes , 2004, Scientometrics.

[102]  C. Lee Giles,et al.  Scholarly publishing in the Internet age: a citation analysis of computer science literature , 2001, Inf. Process. Manag..

[103]  Elena Maceviciute Review of: Thelwall, Michael. Introduction to webometrics: quantitative web research for the social sciences. San Rafael, CA: Morgan & Claypool, 2009 , 2010, Inf. Res..

[104]  José Luis Ortega,et al.  Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC's members , 2015, J. Informetrics.

[105]  Mike Thelwall,et al.  Can Mendeley bookmarks reflect readership? A survey of user motivations , 2016, J. Assoc. Inf. Sci. Technol..

[106]  M. Thelwall,et al.  Research Blogs and the Discussion of Scholarly Information , 2012, PloS one.

[107]  Yao-Tsu Tung A Case Study of Undergraduate Course Syllabi in Taiwan , 2010 .

[108]  Alison J. Head,et al.  How Today's College Students use Wikipedia for Course-related Research , 2010, First Monday.

[109]  Tara L. Pummer,et al.  Reliability of Wikipedia as a medication information source for pharmacy students , 2011 .

[110]  Panayiotis Zaphiris,et al.  Cultural Differences in Collaborative Authoring of Wikipedia , 2006, J. Comput. Mediat. Commun..

[111]  Richard E. West,et al.  Mendeley: Creating Communities of Scholarly Inquiry Through Research Collaboration , 2011 .

[112]  Peter E. Doolittle,et al.  Recommended Syllabus Components: What Do Higher Education Faculty Include in Their Syllabi?. , 2010 .

[113]  Mike Thelwall,et al.  Are scholarly articles disproportionately read in their own country? An analysis of mendeley readers , 2015, J. Assoc. Inf. Sci. Technol..

[114]  Kimberly S. Hamilton,et al.  The increasing linkage between U.S. technology and public science , 1997 .

[115]  Teresa Welsh An Evaluation of Online Syllabi in The University of Tennessee College of Communications 1 , 2008 .

[116]  Scott Chamberlain,et al.  Consuming Article-Level Metrics: Observations and Lessons , 2013 .

[117]  Mike Thelwall,et al.  Motivations for URL citations to open access library and information science articles , 2006, Scientometrics.

[118]  Johan Bollen,et al.  How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations , 2012, PloS one.