Relations Between Relevance Assessments, Bibliometrics and Altmetrics

Relevance assessment in retrieval test collections and citations/mentions of scientific documents are two different forms of relevance decisions: direct and indirect. To investigate these relations, we combine arXiv data with Web of Science and Altmetrics data. In this new collection, we assess the effect of relevance ratings on measured perception in the form of citations or mentions, likes, tweets, et cetera. The impact of our work is that we could show a relation between direct relevance assessments and indirect relevance signals.

[1]  Tamara Heck,et al.  Performing Informetric Analysis on Information Retrieval Test Collections: Preliminary Experiments in the Physics Domain , 2013, ArXiv.

[2]  Philipp Mayr,et al.  Users are not Influenced by High Impact and Core Journals while Searching , 2018, BIR@ECIR.

[3]  Ellen M. Voorhees,et al.  TREC: Continuing information retrieval's tradition of experimentation , 2007, CACM.

[4]  M. Thelwall,et al.  F 1000 , Mendeley and Traditional Bibliometric Indicators , 2012 .

[5]  Zeljko Carevic,et al.  On the Connection Between Citation-based and Topical Relevance Ranking: Results of a Pretest using iSearch , 2014, BIR@ECIR.

[6]  Dirk Tunger,et al.  Altmetrics Feasibility Study , 2017 .

[7]  Birger Larsen,et al.  Exploiting Citation Contexts for Physics Retrieval , 2015, BIR@ECIR.

[8]  Mette Skov,et al.  Inter and intra-document contexts applied in polyrepresentation for best match IR , 2008, Inf. Process. Manag..

[9]  Rodrigo Costas,et al.  The thematic orientation of publications mentioned on social media: Large-scale disciplinary comparison of social media metrics with citations , 2015, Aslib J. Inf. Manag..

[10]  K. Holmberg,et al.  The Relationship Between Institutional Factors, Citation and Altmetric Counts of Publications from Finnish Universities , 2019, Journal of Altmetrics.

[11]  Cassidy R. Sugimoto,et al.  Do Altmetrics Work? Twitter and Ten Other Social Web Services , 2013, PloS one.

[12]  H. D. White.  Relevance theory and citations , 2011 .

[13]  Charles Cole,et al.  A theory of information need for information retrieval that connects information to knowledge , 2011, J. Assoc. Inf. Sci. Technol..

[14]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[15]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[16]  S. Haustein,et al.  Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns , 2015, PloS one.

[17]  Éric Archambault,et al.  Towards a Multilingual, Comprehensive and Open Scientific Journal Ontology , 2013 .