Visual Analytics for Information Retrieval Evaluation Campaigns

Information Retrieval (IR) has been deeply rooted in experimentation since its inception, allowing researchers and developers to understand the behavior and interactions within increasingly complex IR systems, such as web search engines, which have to address ever increasing user needs and support challenging tasks. This paper focuses on the innovative Visual Analytics (VA) approach realized by the Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation (PROMISE) environment, which simplifies and makes more effective the experimental evaluation process by allowing a formal and structured way to explore the complex data set of measures produced along an evaluation campaign. The system uses the result produced within the Conference and Labs of the Evaluation Forum (CLEF) [Cle].

[1]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[2]  Jaegul Choo,et al.  UTOPIAN: User-Driven Topic Modeling Based on Interactive Nonnegative Matrix Factorization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[3]  Sougata Mukherjea,et al.  Visualizing the results of multimedia Web search engines , 1996, Proceedings IEEE Symposium on Information Visualization '96.

[4]  Herbert A. Sturges,et al.  The Choice of a Class Interval , 1926 .

[5]  Giuseppe Santucci,et al.  Harnessing the Scientific Data Produced by the Experimental Evaluation of Search Engines and Information Access Systems Improved Exploitation of Measures and Analyses in Scientic Production Info Rma Tion , 2022 .

[6]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[7]  Xiaojun Yuan,et al.  Visualizing and mapping the intellectual structure of information retrieval , 2012, Inf. Process. Manag..

[8]  Jin Zhang.  Visualization for Information Retrieval , 2018, Encyclopedia of Database Systems.

[9]  Matthew O. Ward,et al.  XmdvTool: integrating multiple methods for visualizing multivariate data , 1994, Proceedings Visualization '94.

[10]  Donna Harman,et al.  Information Retrieval Evaluation , 2011, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[11]  Giuseppe Santucci,et al.  Visual Analytics and Information Retrieval , 2012, PROMISE Winter School.

[12]  James J. Thomas,et al.  Visualizing the non-visual: spatial analysis and interaction with information from text documents , 1995, Proceedings of Visualization 1995 Conference.

[13]  Jarke J. van Wijk,et al.  Flexible Linked Axes for Multivariate Data Visualization , 2011, IEEE Transactions on Visualization and Computer Graphics.

[14]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[15]  Nick Cramer,et al.  WebTheme™: Understanding Web Information through Visual Analytics , 2002, International Semantic Web Conference.