Human-Computer Interaction View on Information Retrieval Evaluation

The field of information retrieval (IR) has experienced tremendous growth over the years. Researchers have however identified Human-Computer Interaction (HCI) aspects as important concerns in IR research. Incorporation of HCI techniques in IR can ensure that IR systems intended for human users are developed and evaluated in a way that is consistent with and reflects the needs of those users. The traditional methods of evaluating IR systems have for a long period been largely concerned with system-oriented measurements such as precision and recall, but not on the usability aspects of the IR system. There also are no well-established evaluation approaches for studying users and their interactions with IR systems. This chapter describes the role and place of HCI toward supporting and appropriating the evaluation of IR systems.

[1]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[2]  Donald H. Kraft,et al.  Measurement in Information Science , 1994 .

[3]  Mika Käki,et al.  Controlling the complexity in comparing search user interfaces via user studies , 2008, Information Processing & Management.

[4]  Frances C. Johnson,et al.  Task dimensions of user evaluations of information retrieval systems , 2003, Inf. Res..

[5]  Allison Druin,et al.  Supporting elementary-age children's searching and browsing: Design and evaluation using the international children's digital library , 2007 .

[6]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[7]  Matthew A. Lievesley,et al.  Surrogate users: a pragmatic approach to defining user needs , 2007, CHI Extended Abstracts.

[8]  M. Csíkszentmihályi.  Finding Flow: The Psychology of Engagement with Everyday Life , 1997 .

[9]  Stephen P. Harter,et al.  Online Information Retrieval: Concepts, Principles and Techniques , 1986 .

[10]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[11]  Gregory D. Abowd,et al.  Human-Computer Interaction (3rd Edition) , 2003 .

[12]  Tefko Saracevic,et al.  Modeling Interaction in Information Retrieval (IR): A Review and Proposal. , 1996 .

[13]  Gerard Salton,et al.  Evaluation problems in interactive information retrieval , 1969, Inf. Storage Retr..

[14]  Gerard Salton,et al.  Improving Retrieval Performance by Relevance Feedback , 1997 .

[15]  Brant C. White,et al.  United States patent , 1985 .

[16]  Mark V. Springett,et al.  A comparison of usability techniques for evaluating design , 1997, DIS '97.

[17]  Tefko Saracevic,et al.  The Stratified Model of Information Retrieval Interaction: Extension and Applications , 1997 .

[18]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[19]  Paul B. Kantor,et al.  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* , 1988 .

[20]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[21]  Nicholas J. Belkin,et al.  A Model for Evaluation of Interactive Information Retrieval , 2009 .

[22]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[23]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[24]  David Miller,et al.  The role of individual differences in Internet searching: an empirical study , 2001 .

[25]  Bryce Allen,et al.  Evaluation of public services and public services personnel , 1991 .

[26]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[27]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[28]  Charles Oppenheim,et al.  A user-centred design and evaluation of IR interfaces , 2006, J. Libr. Inf. Sci..

[29]  Christine L. Borgman,et al.  Children's Searching Behavior on Browsing and Keyword Online Catalogs: The Science Library Catalog Project , 1995, J. Am. Soc. Inf. Sci..

[30]  Aravindan Veerasamy,et al.  Effectiveness of a graphical display of retrieval results , 1997, SIGIR '97.

[31]  RANIA SIATRI,et al.  The Evolution of User Studies , 1999 .

[32]  Peter Ingwersen,et al.  Measures of relative relevance and ranked half-life: performance indicators for interactive IR , 1998, SIGIR '98.

[33]  Diane Kelly,et al.  Questionnaire mode effects in interactive information retrieval experiments , 2008, Inf. Process. Manag..

[34]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[35]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[36]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[37]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[38]  Preben Hansen,et al.  Evaluation of IR User interface – Implications for User Interface Design , 1998 .

[39]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[40]  Bryce Allen,et al.  Information Tasks: Toward a User-Centered Approach to Information Systems , 1996 .

[41]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[42]  James Allan,et al.  Meeting of the MINDS: an information retrieval research agenda , 2007, SIGF.

[43]  Nicholas J. Belkin,et al.  Evaluation of a tool for visualization of information retrieval results , 1996, SIGIR '96.

[44]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[45]  Robert M. Losee.  Evaluating retrieval performance given database and query characteristics: analytic determination of performance surfaces , 1996 .

[46]  D. J. Urquhart.  THE DISTRIBUTION AND USE OF SCIENTIFIC AND TECHNICAL INFORMATION , 1948 .

[47]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[48]  E. Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[49]  Ernest R. House,et al.  Philosophy of Evaluation , 1983 .

[50]  Jean Tague-Sutcliffe,et al.  Measuring information : an information services perspective , 1995 .

[51]  VeerasamyAravindan,et al.  Effectiveness of a graphical display of retrieval results , 1997 .

[52]  Jean Tague.  Informativeness as an ordinal utility function for information retrieval , 1987, SIGIR 1987.

[53]  Prudence W. Dalrymple.  Retrieval by reformulation in two library catalogs: Toward a cognitive model of searching behavior , 1990 .

[54]  Deborah Hix,et al.  Developing user interfaces , 1993 .

[55]  David Ellis,et al.  A behavioural model for information retrieval system design , 1989, J. Inf. Sci..

[56]  David Ellis,et al.  Modelling the information seeking patterns of engineers and research scientists in an industrial environment , 1997, J. Documentation.

[57]  Gary Marchionini.  Toward human‐computer information retrieval , 2007 .

[58]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[59]  Charles T. Meadow,et al.  A study of the use of variables in information retrieval user studies , 1999 .

[60]  Tom Carey,et al.  ACM SIGCHI Curricula for Human-Computer Interaction , 1992 .

[61]  Gerard Salton,et al.  The State of Retrieval System Evaluation , 1992, Inf. Process. Manag..

[62]  Peter Willett,et al.  Readings in information retrieval , 1997 .

[63]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[64]  Prudence W. Dalrymple,et al.  User-centered evaluation of information retrieval , 1991 .

[65]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[66]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988 .

[67]  Marti A. Hearst.  Search User Interfaces: Query Reformulation , 2009 .

[68]  Mark D. Dunlop.  Time, relevance and interaction modelling for information retrieval , 1997, SIGIR '97.

[69]  David Ellis,et al.  A Behavioural Approach to Information Retrieval System Design , 1989, J. Documentation.

[70]  Carol H. Fenichel,et al.  Online searching: Measures that discriminate among users with different types of experiences , 1981, J. Am. Soc. Inf. Sci..

[71]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[72]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[73]  Gerard Salton,et al.  The SMART Retrieval System , 1971 .

[74]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[75]  Xia Lin.  Map displays for information retrieval , 1997 .

[76]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..