Experiences with the 2013-2016 CLEF Interactive Information Retrieval Tracks

This paper describes our experiences with the interactive IR tracks organized at CLEF from 2013-2016 and aggregates the lessons learned with each consecutive instance of the lab. We end with a summary of practical insights and lessons for future collaborative interactive IR evaluation exercises and for potential re-use scenarios.

[1]  Elaine Toms,et al.  Building a Common Framework for IIR Evaluation , 2013, CLEF.

[2]  Pertti Vakkari,et al.  A theory of the task-based information retrieval process: a summary and generalisation of a longitudinal study , 2001, J. Documentation.

[3]  Luanne Freund,et al.  Report on the Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2018) , 2018, SIGF.

[4]  Luanne Freund,et al.  Assigning search tasks designed to elicit exploratory search behaviors , 2012, HCIR '12.

[5]  Martin Braschler,et al.  A PROMISE for Experimental Evaluation , 2010, CLEF.

[6]  Paul Solomon,et al.  Looking for Information—A Survey of Research on Information Seeking, Needs, and Behavior , 2003, Information Retrieval.

[7]  David Ellis,et al.  A behavioural model for information retrieval system design , 1989, J. Inf. Sci..

[8]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[9]  Nils Pharo,et al.  Seven Years of INEX Interactive Retrieval Experiments - Lessons and Challenges , 2012, CLEF.

[10]  Nicola Ferro,et al.  CHiC 2011 - Cultural Heritage in CLEF: From Use Cases to Evaluation in Practice for Multilingual Information Access to Cultural Heritage , 2011, CLEF.

[11]  Nicola Ferro,et al.  Cultural Heritage in CLEF (CHiC) Overview 2012 , 2012, CLEF.

[12]  Nicola Ferro,et al.  Cultural Heritage in CLEF (CHiC) , 2013 .

[13]  Nils Pharo,et al.  Examining the effect of task stage and topic knowledge on searcher interaction with a "digital bookstore" , 2012, IIiX.

[14]  Catherine Sheldrick Ross,et al.  Finding without seeking: the information encounter in the context of reading for pleasure , 1999, Inf. Process. Manag..

[15]  Kara Reuter.  Assessing aesthetic relevance: Children's book selection in a digital library , 2007 .

[16]  Elaine Toms,et al.  Overview of the SBS 2015 Interactive Track , 2015, CLEF.

[17]  Elaine Toms,et al.  The development and evaluation of a survey to measure user engagement , 2010, J. Assoc. Inf. Sci. Technol..

[18]  Norbert Fuhr,et al.  Overview and Results of the INEX 2009 Interactive Track , 2010, ECDL.

[19]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[20]  Pia Borlund.  Interactive Information Retrieval: An Evaluation Perspective , 2016, CHIIR.

[21]  Deborah L. Goodall.  Browsing in Public Libraries , 1989 .

[22]  Pertti Vakkari,et al.  A sign of a good book: readers' methods of accessing fiction in the public library , 2013, J. Documentation.

[23]  Jaap Kamps,et al.  Overview of the SBS 2016 Interactive Track , 2016, CLEF.

[24]  Catherine Sheldrick Ross.  Making Choices , 2000 .

[25]  Elaine Toms,et al.  Sheffield Submission to the CHiC Interactive Task: Exploring Digital Cultural Heritage , 2013, CLEF.

[26]  Marijn Koolen,et al.  Overview of the INEX 2014 Interactive Social Book Search Track , 2014, CLEF.

[27]  Elaine Toms,et al.  Cultural Heritage in CLEF (CHiC) 2013 , 2013, CLEF.

[28]  Elaine G. Toms,et al.  A Pluggable Interactive IR Evaluation Work-bench , 2013 .

[29]  Nicholas J. Belkin,et al.  Second Workshop on Supporting Complex Search Tasks , 2017, CHIIR.

[30]  Elaine Toms,et al.  The CHiC Interactive Task (CHiCi) at CLEF2013 , 2013, CLEF.

[31]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[32]  Elaine Toms,et al.  Report on the First Workshop on Supporting Complex Search Tasks , 2015, SIGIR Forum.

[33]  Gabriella Kazai,et al.  Overview of the INEX 2011 Books and Social Search Track , 2011, INEX.

[34]  Chirag Shah,et al.  Collaborative Information Seeking , 2015, Computer Supported Cooperative Work.