Visual Analytics for Information Retrieval Evaluation (VAIRË 2015)

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. The tutorial introduced basic and intermediate concepts about lab-based evaluation of information retrieval systems, its pitfalls, and shortcomings and it complemented them with a recent and innovative angle to evaluation: the application of methodologies and tools coming from the Visual Analytics (VA) domain for better interacting, understanding, and exploring the experimental results and Information Retrieval (IR) system behaviour.

[1]  Nicola Ferro,et al.  CLEF 15th Birthday , 2014, SIGIR Forum.

[2]  Donna Harman,et al.  Information Retrieval Evaluation , 2011, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[3]  Nicole Bauer,et al.  Information Retrieval Implementing And Evaluating Search Engines , 2016 .

[4]  Donna K. Harman,et al.  Overview of the Reliable Information Access Workshop , 2009, Information Retrieval.

[5]  Giuseppe Santucci,et al.  A Visual Interactive Environment for Making Sense of Experimental Data , 2014, ECIR.

[6]  Giuseppe Santucci,et al.  Visual interactive failure analysis: supporting users in information retrieval evaluation , 2012, IIR.

[7]  Giuseppe Santucci,et al.  Information retrieval failure analysis: Visual analytics as a support for interactive “what-if” investigation , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[8]  Martin Braschler,et al.  PROMISE retreat report prospects and opportunities for information access evaluation , 2012, SIGF.

[9]  Nicola Ferro,et al.  CLEF 15th Birthday: What Can We Learn From Ad Hoc Retrieval? , 2014, CLEF.

[10]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[11]  Norman E. Fenton,et al.  Software Metrics: A Rigorous Approach , 1991 .

[12]  John T. Stasko,et al.  Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study , 2009, 2009 IEEE Symposium on Visual Analytics Science and Technology.

[13]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..

[14]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[15]  Giuseppe Santucci,et al.  VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis , 2014, J. Vis. Lang. Comput..

[16]  Daniel A. Keim,et al.  Mastering the Information Age - Solving Problems with Visual Analytics , 2010 .

[17]  James M. Bieman,et al.  Software Metrics: A Rigorous and Practical Approach, Third Edition , 2014 .