Methods for Evaluating Interactive Information Retrieval Systems with Users

This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.

[1]  Larry B. Wallnau,et al.  Statistics for the Behavioral Sciences , 1985 .

[2]  Andrew Turpin,et al.  Do batch and user evaluations give the same results? , 2000, SIGIR '00.

[3]  Joemon M. Jose,et al.  Comparing collaborative and independent search in a recall-oriented task , 2008, IIiX.

[4]  Dag Elgesem,et al.  What is special about the ethical issues in online research? , 2004, Ethics and Information Technology.

[5]  Paul M. Fitts,et al.  Eye movements of aircraft pilots during instrument-landing approaches. , 1950 .

[6]  Gabriella Kazai,et al.  Proceedings of the 2008 ACM Workshop on Research Advances in Large Digital Book Repositories, BooksOnline 2008, Napa Valley, California, USA, October 30, 2008 , 2008, BooksOnline.

[7]  Batya Friedman,et al.  Cookies and Web browser design: toward realizing informed consent online , 2001, CHI.

[8]  Fabio Crestani,et al.  Written versus spoken queries: A qualitative and quantitative comparative analysis , 2006, J. Assoc. Inf. Sci. Technol..

[9]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[10]  Rong Tang,et al.  Towards the Identification of the Optimal Number of Relevance Categories , 1999, J. Am. Soc. Inf. Sci..

[11]  S. Robertson.  The probability ranking principle in IR , 1997 .

[12]  Gary Marchionini.  Toward human‐computer information retrieval , 2007 .

[13]  Michael Burmester,et al.  Hedonic and ergonomic quality aspects determine a software's appeal , 2000, CHI.

[14]  Kristian J. Hammond,et al.  User interactions with everyday applications as context for just-in-time information access , 2000, IUI '00.

[15]  Eytan Adar,et al.  User 4XXXXX9: Anonymizing Query Logs , 2007 .

[16]  Michael A. Shepherd,et al.  A Field Study Characterizing Web-based Information Seeking Tasks , 2022 .

[17]  R. Abelson.  Statistics As Principled Argument , 1995 .

[18]  William Jones.  Personal Information Management , 2007, Annu. Rev. Inf. Sci. Technol..

[19]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[20]  Jean Tague-Sutcliffe,et al.  Some Perspectives on the Evaluation of Information Retrieval Systems , 1996, J. Am. Soc. Inf. Sci..

[21]  Susan T. Dumais,et al.  The Vocabulary Problem in Human-System Communication: an Analysis and a Solution , 1987 .

[22]  Stephen W. Littlejohn.  Theories of Human Communication , 1978 .

[23]  Daniela Petrelli,et al.  On the role of user-centred evaluation in the advancement of interactive information retrieval , 2008, Inf. Process. Manag..

[24]  Mark Baillie,et al.  The relative effects of knowledge, interest and confidence in assessing relevance , 2007, J. Documentation.

[25]  Mu-Hsuan Huang,et al.  The influence of document presentation order and number of documents judged on users' judgments of relevance , 2004, J. Assoc. Inf. Sci. Technol..

[26]  Christine L. Borgman,et al.  All users of information retrieval systems are not created equal: An exploration into individual differences , 1989, Inf. Process. Manag..

[27]  Brenda Dervin,et al.  Given a context by any other name: methodological tools for taming the unruly beast , 1997 .

[28]  Peter Bailey,et al.  Efficient and flexible search using text and metadata , 2000 .

[29]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[30]  Douglas W. Oard.  Evaluating Interactive Cross-Language Information Retrieval: Document Selection , 2000, CLEF.

[31]  Robert M. Losee,et al.  Feedback in Information Retrieval. , 1996 .

[32]  Jean Tague.  Informativeness as an ordinal utility function for information retrieval , 1987, SIGIR 1987.

[33]  Donna K. Harman,et al.  Evaluation Issues in Information Retrieval , 1992, Inf. Process. Manag..

[34]  Madhubalan Viswanathan,et al.  Measurement error and research design , 2005 .

[35]  R E Wood,et al.  Impact of guided exploration and enactive exploration on self-regulatory mechanisms and information acquisition through electronic search. , 2001, The Journal of applied psychology.

[36]  James A. Holstein,et al.  Qualitative Interviewing and Grounded Theory Analysis , 2003 .

[37]  Stefano Mizzaro.  Relevance: the whole history , 1997 .

[38]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[39]  Ayse Göker,et al.  Evaluation of a mobile information system in context , 2008, Inf. Process. Manag..

[40]  Aravindan Veerasamy,et al.  Effectiveness of a graphical display of retrieval results , 1997, SIGIR '97.

[41]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988, J. Am. Soc. Inf. Sci..

[42]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[43]  Chirag Shah,et al.  Effects of performance feedback on users' evaluations of an interactive IR system , 2008, IIiX.

[44]  G. Katona,et al.  The Art of Asking Questions , 1951 .

[45]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[46]  RANIA SIATRI,et al.  The Evolution of User Studies , 1999 .

[47]  Jean Tague-Sutcliffe.  Some perspectives on the evaluation of information retrieval systems , 1996 .

[48]  M. Csíkszentmihályi.  Finding Flow: The Psychology of Engagement with Everyday Life , 1997 .

[49]  R. Amdur,et al.  Institutional Review Board: Member Handbook , 2003 .

[50]  Pertti Vakkari,et al.  Explanation in Information Seeking and Retrieval , 2005 .

[51]  Ninghui Li,et al.  End-User Privacy in Human–Computer Interaction , 2009 .

[52]  Pertti Vakkari,et al.  The influence of relevance levels on the effectiveness of interactive information retrieval , 2004, J. Assoc. Inf. Sci. Technol..

[53]  Peter Ingwersen,et al.  Measures of relative relevance and ranked half-life: performance indicators for interactive IR , 1998, SIGIR '98.

[54]  Mary Czerwinski,et al.  A diary study of task switching and interruptions , 2004, CHI.

[55]  Susan T. Dumais,et al.  Personalizing Search via Automated Analysis of Interests and Activities , 2005, SIGIR.

[56]  Thorsten Joachims,et al.  The influence of task and gender on search and evaluation behavior using Google , 2006, Inf. Process. Manag..

[57]  Peter Bruza,et al.  Web searching: A process-oriented experimental study of three interactive search paradigms , 2002, J. Assoc. Inf. Sci. Technol..

[58]  Falk Scholer,et al.  User performance versus precision measures for simple search tasks , 2006, SIGIR.

[59]  Charles T. Meadow,et al.  A study of the use of variables in information retrieval user studies , 1999 .

[60]  P. Hancock,et al.  Human Mental Workload , 1988 .

[61]  Pertti Vakkari,et al.  Changes in relevance criteria and problem stages in task performance , 2000, J. Documentation.

[62]  Andrew Dillon,et al.  User analysis in HCI - the historical lessons from individual differences research , 1996, Int. J. Hum. Comput. Stud..

[63]  Peiling Wang,et al.  Methodologies and Methods for User Behavioral Research. , 1999 .

[64]  Peter G. Anick.  Using terminological feedback for web search refinement: a log-based study , 2003, SIGIR.

[65]  Pia Borlund,et al.  Experimental components for the evaluation of interactive information retrieval systems , 2000, J. Documentation.

[66]  Jean M. Tague,et al.  The pragmatics of information retrieval experimentation , 1981 .

[67]  Jean Tague-Sutcliffe.  Informativeness as an Ordinal Utility Function for Information Retrieval , 1987, SIGIR Forum.

[68]  Mark S. Ackerman,et al.  The perfect search engine is not enough: a study of orienteering behavior in directed search , 2004, CHI.

[69]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[70]  Benjamin B. Bederson,et al.  Interfaces for staying in the flow , 2004, UBIQ.

[71]  Gerard Salton,et al.  The State of Retrieval System Evaluation , 1992, Inf. Process. Manag..

[72]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[73]  Preben Hansen,et al.  Conceptual framework for tasks in information studies: Book Reviews , 2005 .

[74]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[75]  Micheline Beaulieu.  Interaction in information searching and retrieval , 2000, J. Documentation.

[76]  Jussi Karlgren,et al.  Verbosity and Interface Design , 2000 .

[77]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[78]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[79]  Marija Norvaisaite,et al.  Review of: Fisher, Karen E., Erdelez, Sandra, and McKechnie, Lynne E.F. Theories of information behavior. Medford, NJ: Information Today, Inc. 2005 , 2006, Inf. Res..

[80]  George Buchanan,et al.  The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work , 2008, Inf. Process. Manag..

[81]  P. Willett,et al.  An Introduction to Algorithmic and Cognitive Approaches for Information Retrieval , 1995 .

[82]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[83]  Gary Marchionini,et al.  Evaluating hypermedia and learning: methods and results from the Perseus Project , 1994, TOIS.

[84]  Pertti Vakkari,et al.  Changes in Search Tactics and Relevance Judgements when Preparing a Research Proposal A Summary of the Findings of a Longitudinal Study , 2001, Information Retrieval.

[85]  Heidi E. Julien,et al.  Affective issues in library and information science systems work: A content analysis , 2005 .

[86]  James Allan,et al.  HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents , 2003, TREC.

[87]  Xiangmin Zhang.  Collaborative relevance judgment: A group consensus method for evaluating user search performance , 2002, J. Assoc. Inf. Sci. Technol..

[88]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[89]  Xin Fu,et al.  Elicitation of term relevance feedback: an investigation of term source and context , 2006, SIGIR.

[90]  T. D. Wilson,et al.  On user studies and information needs , 2006, J. Documentation.

[91]  Tefko Saracevic,et al.  The Stratified Model of Information Retrieval Interaction: Extension and Applications , 1997 .

[92]  Hans Peter Luhn,et al.  A Business Intelligence System , 1958, IBM J. Res. Dev..

[93]  Jakob Nielsen,et al.  Measuring usability: preference vs. performance , 1994, CACM.

[94]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[95]  Nicholas J. Belkin,et al.  Interaction in information systems : a review of research from document retrieval to knowledge-based systems , 1985 .

[96]  William Sugar.  User-Centered Perspective of Information Retrieval Research and Analysis Methods. , 1995 .

[97]  Paul Over,et al.  Comparing interactive information retrieval systems across sites: the TREC-6 interactive track matrix experiment , 1998, SIGIR '98.

[98]  Donald O. Case,et al.  Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior , 2012 .

[99]  Mark D. Dunlop,et al.  Exploring the layers of information retrieval evaluation , 1998, Interact. Comput..

[100]  K. Fisher,et al.  Theories of information behavior , 2005 .

[101]  Linda S. Lotto.  Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[102]  Nicholas J. Belkin,et al.  A faceted approach to conceptualizing tasks in information seeking , 2008, Inf. Process. Manag..

[103]  Arne Jönsson,et al.  Wizard of Oz studies -- why and how , 1993, Knowl. Based Syst..

[104]  Nicholas J. Belkin,et al.  Relationships between categories of relevance criteria and stage in task completion , 2007, Inf. Process. Manag..

[105]  Stephen P. Harter,et al.  Evaluation of information retrieval systems : Approaches, issues, and methods , 1997 .

[106]  David Hawking,et al.  Evaluation by comparing result sets in context , 2006, CIKM '06.

[107]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[108]  Barbara M. Wildemuth,et al.  Applications of Social Research Methods to Questions in Information and Library Science , 2009 .

[109]  Dagobert Soergel,et al.  Selecting and measuring task characteristics as independent variables , 2006, ASIST.

[110]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[111]  Brenda Dervin,et al.  From the mind’s eye of the user: The sense-making qualitative-quantitative methodology. , 1992 .

[112]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[113]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988 .

[114]  Colleen Cool.  The Concept of Situation in Information Science. , 2001 .

[115]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[116]  Jeonghyun Kim,et al.  Task as a predictable indicator for information seeking behavior on the Web , 2007 .

[117]  William H. Beyer,et al.  Handbook of Tables for Probability and Statistics , 1967 .

[118]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[119]  R. Fidel.  Qualitative methods in information retrieval research. , 1993 .

[120]  Eric Horvitz,et al.  SearchTogether: an interface for collaborative web search , 2007, UIST.

[121]  David Miller,et al.  Web search strategies and human individual differences: Cognitive and demographic factors, Internet attitudes, and approaches , 2005, J. Assoc. Inf. Sci. Technol..

[122]  Irene Lopatovska,et al.  Willingness to pay and experienced utility as measures of affective value of information objects: Users' accounts , 2008, Inf. Process. Manag..

[123]  KellyDiane.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009 .

[124]  Nicholas J. Belkin,et al.  Interaction in Information Retrieval: Trends Over Time , 1999, J. Am. Soc. Inf. Sci..

[125]  Amanda Spink,et al.  Multiple Search Sessions Model of End-User Behavior: An Exploratory Study , 1996, J. Am. Soc. Inf. Sci..

[126]  J. P. Guilford,et al.  Fundamental statistics in psychology and education , 1943 .

[127]  Sandra G. Hirsh,et al.  Seeking information in order to produce information: An empirical study at Hewlett Packard Labs , 2004, J. Assoc. Inf. Sci. Technol..

[128]  Reijo Savolainen.  Everyday life information seeking: Approaching information seeking in the context of “way of life” , 1995 .

[129]  Nicholas J. Belkin,et al.  Modeling Multiple Information Seeking Episodes. , 2000 .

[130]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[131]  Paul Over,et al.  Interactivity at the Text Retrieval Conference (TREC) , 2001, Inf. Process. Manag..

[132]  Jean Scholtz,et al.  User-Centered Evaluation of Interactive Question Answering Systems , 2006, HLT-NAACL 2006.

[133]  Jacek Gwizdka,et al.  Revisiting search task difficulty: Behavioral and individual difference measures , 2008, ASIST.

[134]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[135]  Lei Wen,et al.  The Effects on Topic Familiarity on Online Search Behaviour and Use of Relevance Criteria , 2006, ECIR.

[136]  J. Guilford.  Fundamental statistics in psychology and education , 1943 .

[137]  Jean Tague-Sutcliffe.  Measuring the informativeness of a retrieval process , 1992, SIGIR '92.

[138]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[139]  R. Riding,et al.  Cognitive Styles—an overview and integration , 1991 .

[140]  James Allan,et al.  Meeting of the MINDS: an information retrieval research agenda , 2007, SIGF.

[141]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1993 .

[142]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[143]  William R. Hersh,et al.  Evaluating Interactive Question Answering , 2008 .

[144]  Michael B. Eisenberg.  Measuring relevance judgments , 1988, Inf. Process. Manag..

[145]  N. Denzin,et al.  Handbook of Qualitative Research , 1994 .

[146]  J. Bradley.  Methodological Issues and Practices in Qualitative Research , 1993, The Library Quarterly.

[147]  D. J. Urquhart.  THE DISTRIBUTION AND USE OF SCIENTIFIC AND TECHNICAL INFORMATION , 1948 .

[148]  S. Deshpande,et al.  Task Characteristics and the Experience of Optimal Flow in Human—Computer Interaction , 1994 .

[149]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[150]  Mark D. Dunlop.  Time, relevance and interaction modelling for information retrieval , 1997, SIGIR '97.

[151]  Evelyn Jacob,et al.  Qualitative Research Traditions: A Review , 1987 .

[152]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[153]  Pertti Vakkari,et al.  Task-based information searching , 2005, Annu. Rev. Inf. Sci. Technol..

[154]  William R. Hersh,et al.  Towards new measures of information retrieval evaluation , 1995, SIGIR '95.

[155]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[156]  Paul B. Kantor,et al.  Cross-Evaluation: A new model for information system evaluation , 2006, J. Assoc. Inf. Sci. Technol..

[157]  Mercer Jennifer Ann,et al.  PUBLICATION manual of the American Psychological Association. , 1952, Psychological bulletin.

[158]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[159]  Eero Sormunen,et al.  Liberal relevance criteria of TREC -: counting on negligible documents? , 2002, SIGIR '02.

[160]  Rosalind W. Picard.  Affective Computing , 1997 .

[161]  E. Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[162]  Sarah Flicker,et al.  Ethical Dilemmas in Research on Internet Communities , 2004, Qualitative health research.

[163]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[164]  Barbara N. Flagg.  Formative Evaluation for Educational Technologies , 1989 .

[165]  Amanda Spink.  Study of interactive feedback during mediated information retrieval , 1997 .

[166]  David Elsweiler,et al.  Towards task-based personal information management evaluations , 2007, SIGIR.

[167]  Diane Kelly,et al.  Questionnaire mode effects in interactive information retrieval experiments , 2008, Inf. Process. Manag..

[168]  Nicholas J. Belkin,et al.  Rutgers Interactive Track at TREC-5 , 1996, TREC.

[169]  Bryce Allen,et al.  Information needs: a person-in-situation approach , 1997 .

[170]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[171]  Christine L. Borgman.  End user behavior on an online information retrieval system: a computer monitoring study , 1983, SIGIR 1983.

[172]  Soo Young Rieh.  Judgement of information quality and cognitive authority in the Web , 2002 .

[173]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[174]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[175]  David J. Pittenger,et al.  Internet Research: An Opportunity to Revisit Classic Ethical Problems in Behavioral Research , 2003, Ethics & behavior.

[176]  Frederick Williams,et al.  Reasoning With Statistics: How To Read Quantitative Research , 1986 .

[177]  Amanda Spink,et al.  Multitasking information seeking and searching processes , 2002, J. Assoc. Inf. Sci. Technol..

[178]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[179]  Ehud Rivlin,et al.  Placing search in context: the concept revisited , 2002, TOIS.

[180]  Paul Solomon,et al.  Discovering information in context , 2005, Annu. Rev. Inf. Sci. Technol..

[181]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[182]  Wallace Koehler,et al.  Information science as "Little Science":The implications of a bibliometric analysis of theJournal of the American Society for Information Science , 2001, Scientometrics.

[183]  Ryen W. White,et al.  Mining the search trails of surfing crowds: identifying relevant websites from user activity , 2008, WWW.

[184]  Ian Ruthven,et al.  Interactive information retrieval , 2008, Annu. Rev. Inf. Sci. Technol..

[185]  Stanley L. Payne,et al.  The Art of Asking Questions , 1951 .

[186]  Elaine Toms,et al.  Developing and evaluating a reliable measure of user engagement , 2008, ASIST.

[187]  Ellen M. Voorhees.  On test collections for adaptive information retrieval , 2008, Inf. Process. Manag..

[188]  Donald H. Kraft,et al.  Experimental and quasi-experimental designs for research in information science , 1984, Inf. Process. Manag..

[189]  Joemon M. Jose,et al.  How users assess Web pages for information seeking , 2005, J. Assoc. Inf. Sci. Technol..

[190]  Peter Bailey,et al.  Relevance assessment: are judges exchangeable and does it matter , 2008, SIGIR '08.

[191]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .

[192]  Ian Ruthven,et al.  Introduction to the special issue on evaluating interactive information retrieval systems , 2008, Inf. Process. Manag..

[193]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[194]  Stephen E. Robertson,et al.  Okapi at TREC-5 , 1996, TREC.

[195]  Zhiwei Guan,et al.  The validity of the stimulated retrospective think-aloud method as measured by eye tracking , 2006, CHI.

[196]  Carol H. Fenichel,et al.  Online searching: Measures that discriminate among users with different types of experiences , 1981, J. Am. Soc. Inf. Sci..

[197]  Paul B. Kantor,et al.  A study of information seeking and retrieving. III. Searchers, searches, and overlap , 1988, J. Am. Soc. Inf. Sci..

[198]  Nigel Ford,et al.  Web search strategies and human individual differences: Cognitive and demographic factors, Internet attitudes, and approaches: Research Articles , 2005 .

[199]  Nicholas J. Belkin,et al.  Evaluation of a tool for visualization of information retrieval results , 1996, SIGIR '96.

[200]  Gerard Salton,et al.  Research and Development in Information Retrieval , 1982, Lecture Notes in Computer Science.

[201]  Louis M. Gomez,et al.  Formative design evaluation of superbook , 1989, TOIS.

[202]  Joemon M. Jose,et al.  Effectiveness of additional representations for the search result presentation on the web , 2008, Inf. Process. Manag..

[203]  J. Walther.  Research ethics in Internet-enabled research: Human subjects issues and methodological myopia , 2002, Ethics and Information Technology.

[204]  Luanne Freund,et al.  Revisiting informativeness as a process measure for information interaction , 2007 .

[205]  Rong Tang,et al.  Towards the Identification of the Optimal Number of Relevance Categories , 1999, J. Am. Soc. Inf. Sci..

[206]  Paul B. Kantor,et al.  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* , 1988 .

[207]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[208]  Jimmy J. Lin,et al.  Overview of the TREC 2007 Question Answering Track , 2008, TREC.

[209]  Gary Marchionini,et al.  Access via Features versus Access via Transcripts: User Performance and Satisfaction , 2003, TRECVID.

[210]  Kerry Rodden,et al.  Exploring How Mouse Movements Relate to Eye Movements on Web Search Results Pages , 2007 .

[211]  Robert M. Losee.  Evaluating retrieval performance given database and query characteristics: analytic determination of performance surfaces , 1996 .

[212]  Kalervo Järvelin,et al.  Assessing learning outcomes in two information retrieval learning environments , 2005, Inf. Process. Manag..

[213]  Beth Sandore,et al.  An introduction to the special section on transaction log analysis , 1993 .

[214]  Mary Czerwinski,et al.  Subjective Duration Assessment: An Implicit Probe for Software Usability , 2001 .

[215]  W. D. Penniman,et al.  Automated monitoring to support the analysis and evaluation of information systems , 1979, SIGIR 1979.

[216]  Thorsten Joachims,et al.  Eye tracking and online search: Lessons learned and challenges ahead , 2008 .

[217]  Eric Brill,et al.  Improving web search ranking by incorporating user behavior information , 2006, SIGIR.

[218]  Ryen W. White,et al.  Studying the use of popular destinations to enhance web search interaction , 2007, SIGIR.

[219]  Amanda Spink,et al.  How are we searching the World Wide Web? A comparison of nine search engine transaction logs , 2006, Inf. Process. Manag..

[220]  Mark Ginsburg,et al.  Client-side monitoring for Web mining , 2003, J. Assoc. Inf. Sci. Technol..

[221]  J. Morahan-Martin.  Males, females, and the Internet. , 1998 .

[222]  Carolyn Watters,et al.  A field study characterizing Web-based information-seeking tasks , 2007 .

[223]  Donna K. Harman,et al.  The TREC Test Collections , 2005 .

[224]  Nicholas J. Belkin,et al.  Interaction in Information Retrieval: Trends Over Time , 1999, J. Am. Soc. Inf. Sci..

[225]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[226]  Raya Fidel,et al.  Online searching styles: A case-study-based model of searching behavior , 1984, J. Am. Soc. Inf. Sci..

[227]  Kalervo Järvelin,et al.  Task complexity affects information seeking and use , 1995 .

[228]  M. Rorvig.  Psychometric measurement and information retrieval , 1988 .

[229]  John E. Leide,et al.  Controlled user evaluations of information visualization interfaces for text retrieval: Literature review and meta-analysis , 2008 .

[230]  Kalervo Järvelin,et al.  An analysis of two approaches in information retrieval: From frameworks to study designs , 2007, J. Assoc. Inf. Sci. Technol..

[231]  Michael D. Heine.  Simulation , and simulation experiments , 2008 .

[232]  Eeva M. Pilke.  Flow experiences in information technology use , 2004, Int. J. Hum. Comput. Stud..

[233]  Susannah R. Stern,et al.  Encountering Distressing Information in Online Research: A Consideration of Legal and Ethical Responsibilities , 2003, New Media Soc..

[234]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[235]  Bernard J. Jansen,et al.  Search log analysis: What it is, what's been done, how to do it , 2006 .

[236]  Amanda Spink,et al.  New Directions in Cognitive Information Retrieval , 2005 .

[237]  Nicholas J. Belkin,et al.  Helping people find what they don't know , 2000, CACM.

[238]  Earl R. Babbie,et al.  The practice of social research , 1969 .

[239]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[240]  Donald H. Kraft,et al.  Measurement in Information Science , 1994 .

[241]  B. Dervin,et al.  Information seeking in context : proceedings of an international conference on research in information needs, seeking and use in different contexts 14-16 August, 1996, Tampere, Finland , 1997 .

[242]  Elaine Toms,et al.  WiIRE: the Web interactive information retrieval experimentation system prototype , 2004, Inf. Process. Manag..

[243]  Rolf T. Wigand,et al.  Exploring Web users' optimal flow experiences , 2000, Inf. Technol. People.

[244]  Jacek Gwizdka,et al.  Personal information management , 2004, CHI EA '04.

[245]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[246]  Jonathan Foster,et al.  Collaborative information seeking and retrieval , 2006, Annu. Rev. Inf. Sci. Technol..

[247]  David Miller,et al.  The role of individual differences in Internet searching: an empirical study , 2001 .

[248]  M. Naaman,et al.  Lost in memories: interacting with photo collections on PDAs , 2004, Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004..

[249]  Mika Käki,et al.  Controlling the complexity in comparing search user interfaces via user studies , 2008, Information Processing & Management.

[250]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[251]  Bernard J. Jansen,et al.  Wrapper: An Application for Evaluating Exploratory Searching Outside of the Lab , 2006 .

[252]  Amanda Spink,et al.  Study of Interactive Feedback During Mediated Information Retrieval , 1997, J. Am. Soc. Inf. Sci..

[253]  Robert M. Losee.  Evaluating Retrieval Performance Given Database and Query Characteristics: Analytic Determination of Performance Surfaces , 1996, J. Am. Soc. Inf. Sci..

[254]  Jerome L. Myers,et al.  Research Design and Statistical Analysis , 1991 .

[255]  Ryen W. White,et al.  Evaluating implicit feedback models using searcher simulations , 2005, TOIS.

[256]  Amanda Spink,et al.  Regions and levels: Measuring and mapping users' relevance judgments , 2001, J. Assoc. Inf. Sci. Technol..

[257]  Nicholas J. Belkin,et al.  Query length in interactive information retrieval , 2003, SIGIR.

[258]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[259]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[260]  Daniel M. Russell,et al.  Query logs alone are not enough , 2007 .

[261]  Andrew Turpin,et al.  Why batch and user evaluations do not give the same results , 2001, SIGIR '01.

[262]  Ian Ruthven,et al.  Searcher's Assessments of Task Complexity for Web Searching , 2004, ECIR.

[263]  Tefko Saracevic.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007 .

[264]  Jereme Haack,et al.  Glass box: capturing, archiving, and retrieving workstation activities , 2006, CARPE '06.

[265]  Katriina Byström,et al.  Information and information sources in tasks of varying complexity , 2002, J. Assoc. Inf. Sci. Technol..

[266]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[267]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[268]  Terry C. Lansdown,et al.  The mind's eye: cognitive and applied aspects of eye movement research , 2005 .

[269]  Ian Ruthven,et al.  Integrating approaches to relevance , 2005 .

[270]  Wendy E. Mackay,et al.  Ethics, lies and videotape… , 1995, CHI '95.

[271]  Amanda Spink,et al.  Regions and levels: Measuring and mapping users' relevance judgments , 2001, J. Assoc. Inf. Sci. Technol..

[272]  Jean Tague-Sutcliffe,et al.  Evaluation of the user interface in an information retrieval system: A model , 1989, Inf. Process. Manag..

[273]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[274]  Preben Hansen,et al.  Conceptual framework for tasks in information studies , 2005, J. Assoc. Inf. Sci. Technol..

[275]  Bryce Allen,et al.  Cognitive and task influences on Web searching behavior , 2002, J. Assoc. Inf. Sci. Technol..

[276]  Soo Young Rieh.  Judgment of information quality and cognitive authority in the Web , 2002, J. Assoc. Inf. Sci. Technol..

[277]  Catherine L. Smith,et al.  User adaptation: good results from poor systems , 2008, SIGIR '08.

[278]  Deborah Compeau,et al.  Computer Self-Efficacy: Development of a Measure and Initial Test , 1995, MIS Q..

[279]  Amanda Spink,et al.  Information seeking and mediated searching. Part 4. Cognitive styles in information seeking , 2002, J. Assoc. Inf. Sci. Technol..

[280]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[281]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[282]  Eugene Agichtein,et al.  Towards Privacy-Preserving Query Log Publishing , 2007 .

[283]  Tom Peters,et al.  The history and development of transaction log analysis , 1993 .

[284]  Elaine Toms,et al.  Task Effects on Interactive Search: The Query Factor , 2008, INEX.

[285]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[286]  Peter Bailey,et al.  Ecient and Flexible Search Using Text and Metadata CSIRO Mathematical and Information Sciences Technical Report 2000/83 , 2000 .

[287]  Elfreda A. Chatman,et al.  The impoverished life‐world of outsiders , 1996 .

[288]  Jean Tague-Sutcliffe,et al.  Measuring information : an information services perspective , 1995 .

[289]  M. Csíkszentmihályi,et al.  Validity and Reliability of the Experience‐Sampling Method , 1987, The Journal of nervous and mental disease.

[290]  Diane Kelly,et al.  Using interview data to identify evaluation criteria for interactive, analytical question-answering systems , 2007 .

[291]  Jimmy J. Lin,et al.  Overview of the TREC 2006 ciQA task , 2007, SIGF.

[292]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[293]  Dee Andy Michel.  What is used during cognitive processing in information retrieval and library searching?: eleven sources of search information , 1994 .

[294]  Douglas W. Oard,et al.  Task-based interaction with an integrated multilingual, multimedia information system: a formative evaluation , 2007, JCDL '07.

[295]  George N. Arnovick,et al.  Design and evaluation of information systems , 1978, Inf. Process. Manag..

[296]  Paul Dourish,et al.  What we talk about when we talk about context , 2004, Personal and Ubiquitous Computing.

[297]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[298]  Jawaid A. Ghani,et al.  The Experience Of Flow In Computer-Mediated And In Face-To-Face Groups , 1991, ICIS.

[299]  Jimmy J. Lin,et al.  How do users find things with PubMed?: towards automatic utility evaluation with user simulations , 2008, SIGIR '08.

[300]  S. Fiske,et al.  Mind the Gap: In Praise of Informal Sources of Formal Theory , 2004, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[301]  Jimmy J. Lin,et al.  What Makes a Good Answer? The Role of Context in Question Answering , 2003, INTERACT.

[302]  Thorsten Joachims,et al.  Accurately interpreting clickthrough data as implicit feedback , 2005, SIGIR '05.

[303]  Gerard Salton,et al.  Evaluation problems in interactive information retrieval , 1969, Inf. Storage Retr..

[304]  Lois M. L. Delcambre,et al.  Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions , 2008, ECIR.

[305]  Henry R. Jex,et al.  Measuring Mental Workload: Problems, Progress, and Promises , 1988 .

[306]  Joemon M. Jose,et al.  Affective feedback: an investigation into the role of emotions in the information seeking process , 2008, SIGIR '08.

[307]  V. Bacharach,et al.  Psychometrics : An Introduction , 2007 .