WiIRE: the Web interactive information retrieval experimentation system prototype

We introduce WiIRE, a prototype system for conducting: interactive information retrieval (IIR) experiments via the Internet. We conceived WiIRE to increase validity while streamlining procedures and adding efficiencies to the conduct of IIR experiments. The system incorporates password-controlled access, online questionnaires, study instructions and tutorials, conditional interface assignment, and conditional query assignment as well as provision for data collection. As an initial evaluation, we used WiIRE inhouse to conduct a Web-based IIR experiment using an external search engine with customized search interfaces and the TREC 11 Interactive Track search queries. Our evaluation of the prototype indicated significant cost efficiencies in the conduct of IIR studies, and additionally had some novel findings about the human perspective: about half participants would have preferred some personal contact with the researcher, and participants spent a significantly decreasing amount of time on tasks over the course of a session.

[1]  Carl Vogel,et al.  Proper methodologies for psychological and sociological studies conducted via the Internet , 1996, Behavior Research Methods, Instruments, & Computers.

[2]  Elaine Toms,et al.  Understanding and facilitating the browsing of electronic text , 2000, Int. J. Hum. Comput. Stud..

[3]  Amanda Spink,et al.  Real life information retrieval: a study of user queries on the Web , 1998, SIGF.

[4]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[5]  M S Horswill,et al.  User-controlled photographic animations, photograph-based questions, and questionnaires: Three Internet-based instruments for measuring drivers’ risk-taking behavior , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[6]  Michael H. Birnbaum,et al.  Introduction to Behavioral Research on the Internet , 2000 .

[7]  Norma Welch,et al.  The World-Wide Web as a medium for psychoacoustical demonstrations and experiments: Experience and results , 1996 .

[8]  Jeffrey Heer,et al.  WebQuilt : A Proxy-based Approach to Remote Web Usability Testing , 2001 .

[9]  Ulf-Dietrich Reips,et al.  Dimensions of Internet science , 2001 .

[10]  Steven D. Penrod,et al.  Methodological variables in Web-based research that may affect results: Sample type, monetary incentives, and personal information , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[11]  Edie Rasmussen,et al.  Evaluating interactive systems in TREC , 1996 .

[12]  James E. Pitkow,et al.  Characterizing Browsing Strategies in the World-Wide Web , 1995, Comput. Networks ISDN Syst..

[13]  Reeshad S. Dalal,et al.  Validity of Web-Based Psychological Research , 2000 .

[14]  William R. Hersh,et al.  An evaluation of interactive Boolean and natural language searching with an online medical textbook , 1995 .

[15]  Alex Kotov.  The Internet as a medium for software engineering experiments , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[16]  Jean Tague-Sutcliffe,et al.  Measuring information : an information services perspective , 1995 .

[17]  Elaine Toms,et al.  Selecting Versus Describing: A Preliminary Analysis of the Efficacy of Categories in Exploring the Web , 2001, TREC.

[18]  Nicholas J. Belkin,et al.  Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval , 2001, Inf. Process. Manag..

[19]  Christoph Hölscher,et al.  Web search behavior of Internet experts and newbies , 2000, Comput. Networks.

[20]  Carol Tenopir,et al.  Users' interaction with World Wide Web resources: an exploratory study using a holistic approach , 2000, Inf. Process. Manag..

[21]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[22]  T. Palys.  Research Decisions, Quantitative and Qualitative Perspectives , 1997 .

[23]  Jacek Gwizdka,et al.  Discriminating Meta-Search: A Framework for Evaluation , 1999, Inf. Process. Manag..

[24]  Ingrid Hsieh-Yee.  Research on Web Search Behavior. , 2001 .

[25]  Brian D. Davison.  Web Traffic Logs: An Imperfect Resource for Evaluation , 1999 .

[26]  Lucas D. Introna,et al.  About experiments and style - A critique of laboratory research in information systems , 2000, Inf. Technol. People.

[27]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[28]  Rudolf Müller,et al.  Design and evaluation of an economic experiment via the Internet , 2001 .

[29]  Ulf-Dietrich Reips,et al.  Financial Incentives, Personal Information and Drop Out in Online Studies , 2001 .

[30]  John H. Krantz,et al.  Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness , 1997 .

[31]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[32]  Dania Bilal,et al.  Differences and similarities in information seeking: children and adults as Web users , 2002, Inf. Process. Manag..

[33]  Jean Tague-Sutcliffe.  Some perspectives on the evaluation of information retrieval systems , 1996 .

[34]  Charles T. Meadow,et al.  A study of the use of variables in information retrieval user studies , 1999 .

[35]  Yin Zhang.  Using the Internet for survey research: a case study , 2000 .

[36]  Ulf-Dietrich Reips,et al.  A Brief History of Web Experimenting , 2000 .

[37]  Elaine Toms,et al.  Augmenting and Limiting Search Queries , 2002, TREC.

[38]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[39]  J Eichstaedt,et al.  An inaccurate-timing filter for reaction time measurement by JAVA applets implementing Internet-based experiments , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[40]  U D Reips.  The Web Experimental Psychology Lab: Five years of data collection on the Internet , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[41]  Alison I. Piper,et al.  Conducting Social Science Laboratory Experiments on the World Wide Web. , 1998 .

[42]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[43]  W C Schmidt.  Presentation accuracy of Web animation methods , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.