Untangling search task complexity and difficulty in the context of interactive information retrieval studies

Purpose – One core element of interactive information retrieval (IIR) experiments is the assignment of search tasks. The purpose of this paper is to provide an analytical review of current practice in developing those search tasks to test, observe or control task complexity and difficulty. Design/methodology/approach – Over 100 prior studies of IIR were examined in terms of how each defined task complexity and/or difficulty (or related concepts) and subsequently interpreted those concepts in the development of the assigned search tasks. Findings – Search task complexity is found to include three dimensions: multiplicity of subtasks or steps, multiplicity of facets, and indeterminability. Search task difficulty is based on an interaction between the search task and the attributes of the searcher or the attributes of the search situation. The paper highlights the anomalies in our use of these two concepts, concluding with suggestions for future methodological research related to search task complexity and d...

[1]  Max L. Wilson,et al.  Information vs interaction: examining different interaction models over consistent metadata , 2012, IIiX.

[2]  Peter Pirolli,et al.  Do your friends make you smarter?: An analysis of social strategies in online information seeking , 2010, Inf. Process. Manag..

[3]  Martin Whittle,et al.  Search engines: a first step to finding information: preliminary findings from a study of observed searches , 2007, Inf. Res..

[4]  Yan Zhang,et al.  The impact of task complexity on people's mental models of MedlinePlus , 2012, Inf. Process. Manag..

[5]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[6]  Yuelin Li,et al.  Exploring the relationships between work task and search task in information search , 2009, J. Assoc. Inf. Sci. Technol..

[7]  M. Iivonen.  Factors lowering the consistency in online searching , 1995 .

[8]  Tony Stockman,et al.  A comparative analysis of the information-seeking behavior of visually impaired and sighted searchers , 2012, J. Assoc. Inf. Sci. Technol..

[9]  Joemon M. Jose,et al.  Slicing and dicing the information space using local contexts , 2006, IIiX.

[10]  Bernard J. Jansen,et al.  Using the taxonomy of cognitive learning to model online searching , 2009, Inf. Process. Manag..

[11]  Craig Locatis,et al.  Searching Through Cyberspace: The Effects of Link Cues and Correspondence on Information Retrieval from Hypertext on the World Wide Web , 1998, J. Am. Soc. Inf. Sci..

[12]  David Elsweiler,et al.  Towards task-based personal information management evaluations , 2007, SIGIR.

[13]  Mirja Iivonen,et al.  Consistency in the Selection of Search Concepts and Search Terms , 1995, Information Processing & Management.

[14]  David Miller,et al.  Web search strategies and retrieval effectiveness: an empirical study , 2002, J. Documentation.

[15]  Jaime Arguello,et al.  Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity , 2012, IIiX.

[16]  Jacek Gwizdka,et al.  What Can Searching Behavior Tell Us About the Difficulty of Information Tasks? A Study of Web Navigation , 2007, ASIST.

[17]  Jacek Gwizdka,et al.  Predicting task difficulty for different task types , 2010, ASIST.

[18]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[19]  J. Chaplin.  Dictionary of psychology , 1975 .

[20]  Nina Wacholder,et al.  User preference: A measure of query-term quality , 2006, J. Assoc. Inf. Sci. Technol..

[21]  Christine L. Borgman.  The user's mental model of an information retrieval system: an experiment on a prototype online catalog , 1999, Int. J. Hum. Comput. Stud..

[22]  Ren Ding,et al.  Assessment of university student web searching competency by a task-based online test: A case study at Wuhan University, China , 2013, Electron. Libr..

[23]  Wendy T. Lucas,et al.  Searching the Web: operator assistance required , 2005, Inf. Process. Manag..

[24]  Elizabeth Foss,et al.  Children's search roles at home: Implications for designers, researchers, educators, and parents , 2012, J. Assoc. Inf. Sci. Technol..

[25]  M. Hakel,et al.  Effects of Objective and Subjective Task Complexity on Performance , 1997 .

[26]  Joseph Sharit,et al.  Investigating the Roles of Knowledge and Cognitive Abilities in Older Adult Information Seeking on the Web , 2008, TCHI.

[27]  Iwan G. J. H. Wopereis,et al.  Differences between novice and experienced users in searching information on the World Wide Web , 2000, J. Am. Soc. Inf. Sci..

[28]  Thura Mack,et al.  Designing for Experts: How Scholars Approach an Academic Library Web Site , 2004 .

[29]  Shelda Debowski,et al.  Wrong way: go back! An exploration of novice search behaviours while conducting an information search , 2001, Electron. Libr..

[30]  Dirk Lewandowski,et al.  Ordinary search engine users assessing difficulty, effort, and outcome for simple and complex search tasks , 2012, IIiX.

[31]  Zhizhong Li,et al.  Task complexity: A review and conceptualization framework , 2012 .

[32]  Dirk Lewandowski,et al.  Ordinary search engine users carrying out complex search tasks , 2012, J. Inf. Sci..

[33]  Mirja Iivonen,et al.  Assessing Level of Difficulty in Web Search Questions , 2002, The Library Quarterly.

[34]  Craig Locatis,et al.  Effects of link annotations on search performance in layered and unlayered hierarchically organized information spaces , 2001, J. Assoc. Inf. Sci. Technol..

[35]  Pia Borlund,et al.  Reconsideration of the simulated work task situation: a context instrument for evaluation of information retrieval interaction , 2010, IIiX.

[36]  Linda C. Smith,et al.  COMPARATIVE ANALYSIS OF ONLINE RETRIEVAL INTERFACES. , 1986 .

[37]  Ian Ruthven,et al.  Searcher's Assessments of Task Complexity for Web Searching , 2004, ECIR.

[38]  Barbara Mirel,et al.  Studying PubMed usages in the field for complex problem solving: Implications for tool design , 2013, J. Assoc. Inf. Sci. Technol..

[39]  D. Campbell.  Task Complexity: A Review and Analysis , 1988 .

[40]  Yvonne Kammerer,et al.  Signpost from the masses: learning effects in an exploratory social tag search browser , 2009, CHI.

[41]  Dan Albertson.  Analyzing user interaction with the ViewFinder video retrieval system , 2010 .

[42]  Joemon M. Jose,et al.  Affective feedback: an investigation into the role of emotions in the information seeking process , 2008, SIGIR '08.

[43]  James Allan,et al.  Predicting searcher frustration , 2010, SIGIR.

[44]  Sandra Goldstein Hirsh.  The effect of domain knowledge on elementary school children's information retrieval behavior on an automated library catalog , 1997 .

[45]  Dania Bilal.  Children's use of the Yahooligans! Web search engine: II. cognitive and physical behaviors on research tasks , 2001 .

[46]  Pertti Vakkari,et al.  Changes in Search Tactics and Relevance Judgements when Preparing a Research Proposal A Summary of the Findings of a Longitudinal Study , 2001, Information Retrieval.

[47]  Jeonghyun Kim,et al.  Describing and predicting information-seeking behavior on the Web , 2009, J. Assoc. Inf. Sci. Technol..

[48]  David Miller,et al.  Web search strategies and human individual differences: Cognitive and demographic factors, Internet attitudes, and approaches , 2005, J. Assoc. Inf. Sci. Technol..

[49]  Pertti Vakkari,et al.  Changes of search terms and tactics while writing a research proposal: A longitudinal case study , 2003, Inf. Process. Manag..

[50]  Gregory K. W. K. Chung,et al.  Children's Internet Searching on Complex Problems: Performance and Process Analyses , 1998, J. Am. Soc. Inf. Sci..

[51]  Daqing He,et al.  Undergraduate students' interaction with online information resources in their academic tasks: A comparative study , 2012, Aslib Proc..

[52]  Stuart K. Card,et al.  The effects of information scent on visual search in the hyperbolic tree browser , 2003, TCHI.

[53]  Jingjing Liu,et al.  Why do users feel search task difficult? , 2013, ASIST.

[54]  Nils Pharo,et al.  The SST Method Schema: a Tool for Analysing Work Task-Based Web Information Search Processes , 2002 .

[55]  Sandra G. Hirsh,et al.  How Do Children Find Information on Different Types of Tasks? Children's Use of the Science Library Catalog , 1997, Libr. Trends.

[56]  Elaine Toms,et al.  Task Effects on Interactive Search: The Query Factor , 2008, INEX.

[57]  Gary Marchionini,et al.  Effects of structure and interaction style on distinct search tasks , 2007, JCDL '07.

[58]  Irene Lopatovska.  Searching for good mood: Examining relationships between search task and mood , 2009, ASIST.

[59]  Luanne Freund,et al.  Documenting and studying the use of assigned search tasks: RepAST , 2014, ASIST.

[60]  Preben Hansen,et al.  Conceptual framework for tasks in information studies , 2005, J. Assoc. Inf. Sci. Technol..

[61]  Alain Breuleux,et al.  A Comparison of Information Retrieval from Print and CD-ROM Versions of an Encyclopedia by Elementary School Students , 1994, Inf. Process. Manag..

[62]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[63]  Pertti Vakkari,et al.  Search effort degrades search output but improves task outcome , 2012, J. Assoc. Inf. Sci. Technol..

[64]  Louis M. Gomez,et al.  Formative design evaluation of superbook , 1989, TOIS.

[65]  Gary Marchionini,et al.  Performance in Electronic Encyclopedias: Implications for Adaptive Systems. , 1991 .

[66]  Judit Bar-Ilan,et al.  The impact of task phrasing on the choice of search keywords and on the search process and success , 2012, J. Assoc. Inf. Sci. Technol..

[67]  Nicholas J. Belkin,et al.  Exploring and predicting search task difficulty , 2012, CIKM '12.

[68]  Marti A. Hearst,et al.  Scatter/gather browsing communicates the topic structure of a very large text collection , 1996, CHI.

[69]  Pertti Vakkari,et al.  Task-based information searching , 2005, Annu. Rev. Inf. Sci. Technol..

[70]  Kalervo Järvelin,et al.  Task complexity affects information seeking and use , 1995 .

[71]  Dan E. Albertson,et al.  Situated topic complexity in interactive video retrieval , 2011, J. Assoc. Inf. Sci. Technol..

[72]  Dave L. Edyburn.  Examining the Successful Retrieval of Information by Students Using Online Databases. , 1988 .

[73]  Karl Gyllstrom,et al.  Undergraduates' evaluations of assigned search topics , 2009, SIGIR.

[74]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..

[75]  Kalervo Järvelin,et al.  Information interaction in molecular medicine: integrated use of multiple channels , 2010, IIiX.