The User in Interactive Information Retrieval Evaluation

This chapter initially defines what characterizes and distinguishes research frameworks from research models. The Laboratory Research Framework for IR illustrates the case. We define briefly what is meant by the concept of research design, including research questions, and what this chapter regards as central IIR evaluation research settings and variables. This is followed by a description of IIR components, pointing to the elements of the Integrated Cognitive Research Framework for IR that incorporates the Laboratory Framework in a contextual manner. The following sections describe and exemplify: (1) Request types, test persons, task-based simulations of search situations and relevance or performance measures in IIR; (2) Ultra-Light Interactive IR experiments; (3) Interactive-Light IR studies; and (4) Naturalistic field investigations of IIR. The chapter concludes with a summary section, a reference list and a thematically classified bibliography.

[1]  Micheline Beaulieu.  Interaction in information searching and retrieval , 2000, J. Documentation.

[2]  Nicholas J. Belkin,et al.  A case for interaction: a study of interactive information retrieval behavior and effectiveness , 1996, CHI.

[3]  Ingrid Hsieh-Yee,et al.  Search Tactics of Web Users in Searching for Texts, Graphics, Known Items and Subjects , 1998 .

[4]  Carol Tenopir,et al.  Users' interaction with World Wide Web resources: an exploratory study using a holistic approach , 2000, Inf. Process. Manag..

[5]  Kalervo Järvelin,et al.  Task complexity affects information seeking and use , 1995 .

[6]  Nicholas J. Belkin,et al.  Distributed expert problem treatment as a model for information system analysis and design , 1982 .

[7]  Xin Fu,et al.  Eliciting better information need descriptions from users of information search systems , 2007, Inf. Process. Manag..

[8]  Gerhard Lakemeyer,et al.  Exploring artificial intelligence in the new millennium , 2003 .

[9]  Jean Tague-Sutcliffe,et al.  Some measures and procedures for evaluation of the user interface in an information retrieval system , 1988, SIGIR '88.

[10]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[11]  Daniela Petrelli,et al.  Observing users, designing clarity: A case study on the user-centered design of a cross-language information retrieval system , 2004, J. Assoc. Inf. Sci. Technol..

[12]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988 .

[13]  Johanna D. Moore,et al.  Proceedings of the Conference on Human Factors in Computing Systems , 1989 .

[14]  Marianne Lykke,et al.  A framework for work task based thesaurus design , 2001, J. Documentation.

[15]  Mark Sanderson,et al.  Information retrieval system evaluation: effort, sensitivity, and reliability , 2005, SIGIR '05.

[16]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[17]  Raya Fidel,et al.  Ranking expansion terms using partial and ostensive evidence , 2002 .

[18]  Kalervo Järvelin.  An analysis of two approaches in information retrieval: From frameworks to study designs , 2007 .

[19]  Samuel Kaski,et al.  Combining eye movements and collaborative filtering for proactive information retrieval , 2005, SIGIR '05.

[20]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[21]  S. Hirsh.  Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[22]  Ian Smith,et al.  Taking email to task: the design and evaluation of a task management centered email tool , 2003, CHI '03.

[23]  Kyung-Sun Kim,et al.  Cognitive style and on-line database search experience as predictors of Web search performance , 2000, J. Am. Soc. Inf. Sci..

[24]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[25]  R. Fidel.  Qualitative methods in information retrieval research. , 1993 .

[26]  Dania Bilal,et al.  Children's use of the Yahooligans! Web search engine. III. Cognitive and physical behaviors on fully self-generated search tasks , 2002, J. Assoc. Inf. Sci. Technol..

[27]  Mark Magennis,et al.  The potential and actual effectiveness of interactive query expansion , 1997, SIGIR '97.

[28]  Janet M. Corrigan,et al.  Background and Methodology , 2000 .

[29]  Chava Nachmias,et al.  Research Methods in the Social Sciences , 1976 .

[30]  W. Bruce Croft,et al.  I 3 R: a new approach to the design of document retrieval systems , 1987 .

[31]  Paul B. Kantor,et al.  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* , 1988 .

[32]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[33]  Peter Bailey,et al.  Overview of the TREC-8 Web Track , 2000, TREC.

[34]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[35]  Mark S. Ackerman,et al.  The perfect search engine is not enough: a study of orienteering behavior in directed search , 2004, CHI.

[36]  Ryen W. White,et al.  An implicit feedback approach for interactive information retrieval , 2006, Inf. Process. Manag..

[37]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[38]  Ryen W. White.  Using searcher simulations to redesign a polyrepresentative implicit feedback interface , 2006, Inf. Process. Manag..

[39]  Peter Ingwersen,et al.  ON THE HOLISTIC COGNITIVE THEORY FOR INFORMATION RETRIEVAL Drifting Outside the Border of the Laboratory Framework , 2007 .

[40]  Jaana Kekäläinen,et al.  Binary and graded relevance in IR evaluations--Comparison of the effects on ranking of IR systems , 2005, Inf. Process. Manag..

[41]  Marianne Lykke Nielsen.  A framework for work task based thesaurus design , 2001 .

[42]  C. J. van Rijsbergen,et al.  Incorporating user search behavior into relevance feedback , 2003, J. Assoc. Inf. Sci. Technol..

[43]  Joemon M. Jose,et al.  Searchers' criteria For assessing web pages , 2003, SIGIR '03.

[44]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[45]  Peiling Wang.  Users' information needs at different stages of a research project: a cognitive view , 1997 .

[46]  Nicholas J. Belkin,et al.  Ask for Information Retrieval: Part II. Results of a Design Study , 1982, J. Documentation.

[47]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[48]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[49]  Ryen W. White,et al.  Evaluating implicit feedback models using searcher simulations , 2005, TOIS.

[50]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[51]  Peter G. Anick.  Using terminological feedback for web search refinement: a log-based study , 2003, SIGIR.

[52]  Pia Borlund,et al.  Experimental components for the evaluation of interactive information retrieval systems , 2000, J. Documentation.

[53]  BilalDania.  Children's use of the Yahooligans! Web search engine , 2001 .

[54]  Ryen W. White,et al.  A task-oriented study on the influencing effects of query-biased summarisation in web searching , 2003, Inf. Process. Manag..

[55]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[56]  Samuel Kaski,et al.  Can Relevance be Inferred from Eye Movements in Information Retrieval , 2003 .

[57]  Ryen W. White,et al.  A study of factors affecting the utility of implicit relevance feedback , 2005, SIGIR '05.

[58]  Micheline Beaulieu,et al.  Experiments on interfaces to support query expansion , 1997, J. Documentation.

[59]  Peter Ingwersen,et al.  Dimensions of relevance , 2000, Inf. Process. Manag..

[60]  Iain Campbell,et al.  Interactive Evaluation of the Ostensive Model Using a New Test Collection of Images with Multiple Relevance Assessments , 2000, Information Retrieval.

[61]  Peter Ingwersen,et al.  Measures of relative relevance and ranked half-life: performance indicators for interactive IR , 1998, SIGIR '98.

[62]  Efthimis N. Efthimiadis,et al.  A user-centred evaluation of ranking algorithms for interactive query expansion , 1993, SIGIR.

[63]  Peter Bruza,et al.  Web searching: A process-oriented experimental study of three interactive search paradigms , 2002, J. Assoc. Inf. Sci. Technol..

[64]  Micheline Hancock-Beaulieu,et al.  Interactive searching and interface issues in the Okapi best match probabilistic retrieval system , 1998, Interact. Comput..

[65]  William R. Hersh,et al.  A task-oriented approach to information retrieval evaluation , 1996 .

[66]  Xin Fu,et al.  The loquacious user: a document-independent source of terms for query expansion , 2005, SIGIR '05.

[67]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[68]  C. J. van Rijsbergen,et al.  Finding Out About: A Cognitive Perspective on Search Engine Technology and the WWW , 2001 .

[69]  David Miller,et al.  The role of individual differences in Internet searching: an empirical study , 2001 .

[70]  Vivian Cothey,et al.  A longitudinal study of World Wide Web users' information-searching behavior , 2002, J. Assoc. Inf. Sci. Technol..

[71]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[72]  Eero Sormunen,et al.  Liberal relevance criteria of TREC -: counting on negligible documents? , 2002, SIGIR '02.

[73]  Miles Efron,et al.  Using Multiple Query Aspects to Build Test Collections without Human Relevance Judgments , 2009, ECIR.

[74]  Pertti Vakkari,et al.  Changes in relevance criteria and problem stages in task performance , 2000, J. Documentation.

[75]  Nicholas J. Belkin,et al.  Display time as implicit feedback: understanding task effects , 2004, SIGIR '04.

[76]  Ivan Koychev,et al.  Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling , 2004, Information Retrieval.

[77]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[78]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[79]  P. Willett,et al.  An Introduction to Algorithmic and Cognitive Approaches for Information Retrieval , 1995 .

[80]  Pertti Vakkari,et al.  The influence of relevance levels on the effectiveness of interactive information retrieval , 2004, J. Assoc. Inf. Sci. Technol..

[81]  Mary Czerwinski,et al.  Subjective Duration Assessment: An Implicit Probe for Software Usability , 2001 .

[82]  Nicholas J. Belkin,et al.  The effect multiple query representations on information retrieval system performance , 1993, SIGIR.

[83]  Mika Käki,et al.  Proportional search interface usability measures , 2004, NordiCHI '04.

[84]  Raya Fidel,et al.  Factors affecting online bibliographic retrieval: A conceptual framework for research , 1983, J. Am. Soc. Inf. Sci..

[85]  Peter Ingwersen,et al.  Searchers' relevance judgments and criteria in evaluating web pages in a learning style perspective , 2008, IIiX.

[86]  Nicholas J. Belkin,et al.  Query length in interactive information retrieval , 2003, SIGIR.

[87]  Marianne Lykke Nielsen.  Task‐based evaluation of associative thesaurus in real‐life environment , 2005 .

[88]  Jenny K. Holder,et al.  A visit to the information mall: Web searching behavior of high school students , 1999 .