Towards a Living Lab for Information Retrieval Research and Development - A Proposal for a Living Lab for Product Search Tasks

The notion of having a “living lab” to undertaken evaluations has been proposed by a number of proponents within the field of Information Retrieval (IR). However, what such a living lab might look like and how it might be setup has not been discussed in detail. Living labs have a number of appealing points such as realistic evaluation contexts where tasks are directly linked to user experience and the closer integration of research/academia and development/industry facilitating more efficient knowledge transfer. However, operationalizing a living lab opens up a number of concerns regarding security, privacy, etc. as well as challenges regarding the design, development and maintenance of the infrastructure required to support such evaluations. Here, we aim to further the discussion on living labs for IR evaluation and propose one possible architecture to create such an evaluation environment. To focus discussion, we put forward a proposal for a living lab on product search tasks within the context of an online shop.

[1]  Nicholas J. Belkin,et al.  Some(what) grand challenges for information retrieval , 2008, SIGF.

[2]  KellyDiane.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009 .

[3]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[4]  Peter A. Todd,et al.  Consumer Reactions to Electronic Shopping on the World Wide Web , 1996, Int. J. Electron. Commer..

[5]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[6]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..

[7]  James Allan,et al.  HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents , 2003, TREC.

[8]  Peter Pirolli.  Powers of 10: Modeling Complex Information-Seeking Systems at Multiple Scales , 2009, Computer.

[9]  Susan T. Dumais,et al.  Evaluation Challenges and Directions for Information-Seeking Support Systems , 2009, Computer.

[10]  Gary Marchionini,et al.  Report on ACM SIGIR 2006 workshop on evaluating exploratory search systems , 2006, SIGF.

[11]  Ron Kohavi,et al.  Responsible editor: R. Bayardo. , 2022 .

[12]  Andrei Z. Broder,et al.  To swing or not to swing: learning when (not) to advertise , 2008, CIKM '08.

[13]  Mark D. Smucker,et al.  Report on the SIGIR 2010 workshop on the simulation of interaction , 2011, SIGF.

[14]  Carol Peters,et al.  Report on the SIGIR 2009 workshop on the future of IR evaluation , 2009, SIGF.

[15]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..