VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis

Objective: Information Retrieval (IR) is strongly rooted in experimentation where new and better ways to measure and interpret the behavior of a system are key to scientific advancement. This paper presents an innovative visualization environment: Visual Information Retrieval Tool for Upfront Evaluation (VIRTUE), which eases and makes more effective the experimental evaluation process. Methods: VIRTUE supports and improves performance analysis and failure analysis. Performance analysis: VIRTUE offers interactive visualizations based on well-known IR metrics allowing us to explore system performances and to easily grasp the main problems of the system. Failure analysis: VIRTUE develops visual features and interaction, allowing researchers and developers to easily spot critical regions of a ranking and grasp possible causes of a failure. Results: VIRTUE was validated through a user study involving IR experts. The study reports on (a) the scientific relevance and innovation and (b) the comprehensibility and efficacy of the visualizations. Conclusion: VIRTUE eases the interaction with experimental results, supports users in the evaluation process and reduces the user effort. Practice: VIRTUE will be used by IR analysts to analyze and understand experimental results. Implications: VIRTUE improves the state-of-the-art in the evaluation practice and integrates visualization and IR research fields in an innovative way.

[1]  Weiguo Fan,et al.  A generic ranking function discovery framework by genetic programming for information retrieval , 2004, Inf. Process. Manag..

[2]  Tetsuya Sakai,et al.  Evaluating evaluation metrics based on the bootstrap , 2006, SIGIR.

[3]  Tetsuya Sakai,et al.  On the reliability of information retrieval metrics based on graded relevance , 2007, Inf. Process. Manag..

[4]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[5]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[6]  Xiaohui Liu,et al.  Editorial: Data mining for understanding user needs , 2010, TCHI.

[7]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[8]  M. Kendall.  Rank Correlation Methods , 1949 .

[9]  Giuseppe Santucci,et al.  Visual interactive failure analysis: supporting users in information retrieval evaluation , 2012, IIR.

[10]  Irving I. Gottesman,et al.  Blind Men and Elephants , 1996 .

[11]  Wendy A. Lawrence-Fowler,et al.  Integrating query thesaurus, and documents through a common visual representation , 1991, SIGIR '91.

[12]  Giuseppe Santucci,et al.  Harnessing the Scientific Data Produced by the Experimental Evaluation of Search Engines and Information Access Systems Improved Exploitation of Measures and Analyses in Scientic Production Info Rma Tion , 2022 .

[13]  Howard D. Wactlar,et al.  Constant density displays using diversity sampling , 2003, IEEE Symposium on Information Visualization 2003 (IEEE Cat. No.03TH8714).

[14]  Jacques Savoy.  Why do successful search systems fail for some topics , 2007, SAC '07.

[15]  Eero Sormunen,et al.  Query performance analyser -: a web-based tool for IR research and instruction , 2002, SIGIR '02.

[16]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[17]  Allan Hanbury,et al.  Patent Retrieval , 2013, Found. Trends Inf. Retr..

[18]  Giuseppe Santucci,et al.  Collaborative Environment of the PROMISE Infrastructure: an "ELEGantt" Approach , 2012, EuroHCIR.

[19]  Donna K. Harman.  Some thoughts on failure analysis for noisy data , 2008, AND '08.

[20]  Sherry Koshman.  Testing user interaction with a prototype visualization-based information retrieval system , 2005, J. Assoc. Inf. Sci. Technol..

[21]  Jin Zhang.  TOFIR: A tool of facilitating information retrieval - introduce a visual retrieval model , 2001, Inf. Process. Manag..

[22]  J. Tukey,et al.  Variations of Box Plots , 1978 .

[23]  Donna K. Harman,et al.  Overview of the Reliable Information Access Workshop , 2009, Information Retrieval.

[24]  Chris Buckley,et al.  Why current IR engines fail , 2004, SIGIR '04.

[25]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[26]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[27]  Jaana Kekäläinen,et al.  Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance , 2008, SIGIR '08.

[28]  Edward A. Fox,et al.  Theoretical Foundations for Digital Libraries: The 5S (Societies, Scenarios, Spaces, Structures, Streams) Approach , 2012, Theoretical Foundations for Digital Libraries.

[29]  Paul Over,et al.  Blind Men and Elephants: Six Approaches to TREC data , 1999, Information Retrieval.

[30]  Jaana Kekäläinen,et al.  Binary and graded relevance in IR evaluations--Comparison of the effects on ranking of IR systems , 2005, Inf. Process. Manag..

[31]  Alan F. Smeaton,et al.  Multilingual and Multimodal Information Access Evaluation, International Conference of the Cross-Language Evaluation Forum, CLEF 2010, Padua, Italy, September 20-23, 2010. Proceedings , 2010, CLEF.

[32]  Donna Harman,et al.  Information Retrieval Evaluation , 2011, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[33]  Martin Braschler,et al.  PROMISE retreat report prospects and opportunities for information access evaluation , 2012, SIGF.

[34]  Giuseppe Santucci,et al.  To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma? , 2011, CLEF.

[35]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[36]  Jin Zhang.  Visualization for Information Retrieval , 2008, The Information Retrieval Series.

[37]  Ellen M. Voorhees,et al.  Evaluating Evaluation Measure Stability , 2000, SIGIR 2000.

[38]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[39]  Nicole Bauer,et al.  Information Retrieval Implementing And Evaluating Search Engines , 2016 .

[40]  Susan T. Dumais,et al.  Potential for personalization , 2010, TCHI.

[41]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[42]  Jacques Savoy,et al.  Statistical inference in retrieval effectiveness evaluation , 1997, Inf. Process. Manag..

[43]  Anja van der Lans.  Enterprise Search and Retrieval (ESR): The Binding Factor , 2013 .

[44]  Ian H. Witten,et al.  How to Build a Digital Library, Second Edition , 2009 .

[45]  Fatima Fahimnia,et al.  Expertise Retrieval: Foundations, Methods and Models , 2016 .

[46]  Ben Carterette,et al.  On rank correlation and the distance between rankings , 2009, SIGIR.

[47]  Giuseppe Santucci,et al.  Cumulated Relative Position: A Metric for Ranking Evaluation , 2012, IIR.

[48]  Marti A. Hearst.  'Natural' search user interfaces , 2011, CACM.

[49]  John W. Tukey,et al.  Exploratory Data Analysis. , 1979 .

[50]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[51]  Giuseppe Santucci,et al.  Visual Comparison of Ranked Result Cumulated Gains , 2011, EuroVA@EuroVis.

[52]  M. de Rijke,et al.  Expertise Retrieval , 2012, Found. Trends Inf. Retr..

[53]  Stephen Robertson,et al.  The methodology of information retrieval experiment , 1981 .

[54]  Ben Shneiderman,et al.  A Rank-by-Feature Framework for Interactive Exploration of Multidimensional Data , 2005, Inf. Vis..

[55]  Ian H. Witten,et al.  How to Build a Digital Library , 2002 .

[56]  Giuseppe Santucci,et al.  Improving Ranking Evaluation Employing Visual Analytics , 2013, CLEF.

[57]  Ellen M. Voorhees,et al.  The seventh text REtrieval conference (TREC-7) , 1999 .

[58]  Djoerd Hiemstra,et al.  Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics , 2012, Lecture Notes in Computer Science.

[59]  Nicola Ferro,et al.  6 – Towards an infrastructure for digital library performance evaluation , 2009 .

[60]  Eero Sormunen,et al.  Liberal relevance criteria of TREC -: counting on negligible documents? , 2002, SIGIR '02.

[61]  Sherry Koshman.  Testing user interaction with a prototype visualization-based information retrieval system: Research Articles , 2005 .

[62]  Giuseppe Santucci,et al.  Interactive Analysis and Exploration of Experimental Evaluation Results , 2011, EuroHCIR.

[63]  Mark Sanderson,et al.  Information retrieval system evaluation: effort, sensitivity, and reliability , 2005, SIGIR '05.

[64]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[65]  Adolfo Jonathan Salinas-López,et al.  CEUR Workshop Proceedings , 2015 .

[66]  Nicola Ferro,et al.  DIRECTions: Design and Specification of an IR Evaluation Infrastructure , 2012, CLEF.

[67]  Mark Levene,et al.  Search Engines: Information Retrieval in Practice , 2011, Comput. J..

[68]  Karel Vredenburg,et al.  A survey of user-centered design practice , 2002, CHI.

[69]  Stefano Mizzaro.  Relevance: the whole history , 1997 .

[70]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[71]  Giorgio Maria Di Nunzio,et al.  10 Years of CLEF Data in DIRECT: Where We Are and Where We Can Go , 2010, EVIA@NTCIR.

[72]  Ellen M. Voorhees,et al.  Evaluation by highly relevant documents , 2001, SIGIR '01.

[73]  Fabio Crestani,et al.  A graphical user interface for the retrieval of hierarchically structured documents , 2004, Inf. Process. Manag..

[74]  Michael Lewis,et al.  Testing visual information retrieval methodologies case study: Comparative analysis of textual, icon, graphical, and spring displays , 2002, J. Assoc. Inf. Sci. Technol..

[75]  Daniel A. Keim,et al.  Visual Comparison of Orderings and Rankings , 2013, EuroVA@EuroVis.

[76]  Gerard Salton,et al.  The SMART Retrieval System , 1971 .