Tutorial on Task-Based Search and Assistance

While great strides are made in the field of search and recommendation, there are still challenges and opportunities to address information access issues that involve solving tasks and accomplishing goals for a wide variety of users. Specifically, we lack intelligent systems that can detect not only the request an individual is making (what), but also understand and utilize the intention (why) and strategies (how) while providing information. Many scholars in the fields of information retrieval, recommender systems, productivity (especially in task management and time management), and artificial intelligence have recognized the importance of extracting and understanding people's tasks and the intentions behind performing those tasks in order to serve them better. However, we are still struggling to support them in task completion, e.g., in search and assistance, it has been challenging to move beyond single-query or single-turn interactions. The proliferation of intelligent agents has opened up new modalities for interacting with information, but these agents will need to be able to work more intelligently in understanding the context and helping the users at task level. This tutorial will introduce the attendees to the issues of detecting, understanding, and using task and task-related information in an information episode (with or without active searching). Specifically, it will cover several recent theories, models, and methods that show how to represent tasks and use behavioral data to extract task information. It will then show how this knowledge or model could contribute to addressing emerging retrieval and recommendation problems.

[1]  Ryen W. White,et al.  Improving searcher models using mouse cursor activity , 2012, SIGIR '12.

[2]  Ryen W. White,et al.  Modeling Long-Term Search Engine Usage , 2010, UMAP.

[3]  Ryen W. White,et al.  Personalized models of search satisfaction , 2013, CIKM.

[4]  Nicholas J. Belkin,et al.  Second Workshop on Supporting Complex Search Tasks , 2017, CHIIR.

[5]  Stefan Ultes,et al.  MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , 2018, EMNLP.

[6]  Kalervo Järvelin,et al.  Search task features in work tasks of varying types and complexity , 2017, J. Assoc. Inf. Sci. Technol..

[7]  Daniel G. Bobrow,et al.  What a to-do: studies of task management towards the design of a personal task list manager , 2004, CHI.

[8]  Ryen W. White,et al.  Task Intelligence Workshop @ WSDM 2019 , 2019, WSDM.

[9]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[10]  HansenPreben,et al.  Conceptual framework for tasks in information studies , 2005 .

[11]  Preben Hansen,et al.  Conceptual framework for tasks in information studies , 2005, J. Assoc. Inf. Sci. Technol..

[12]  Stephen E. Newstead,et al.  The Effects of Prior Experience on Estimating the Duration of Simple Tasks. , 2004 .

[13]  Ryen W. White,et al.  Predicting short-term interests using activity-based search context , 2010, CIKM.

[14]  Ryen W. White,et al.  Task Duration Estimation , 2019, WSDM.

[15]  JärvelinKalervo,et al.  Task complexity affects information seeking and use , 1995 .

[16]  Yang Song,et al.  A task level metric for measuring web search satisfaction and its application on improving relevance estimation , 2011, CIKM '11.

[17]  Ryen W. White,et al.  Modeling dwell time to predict click-level satisfaction , 2014, WSDM.

[18]  Diane Kelly,et al.  The effects of topic familiarity on information search behavior , 2002, JCDL '02.

[19]  Charles L. A. Clarke,et al.  Time-based calibration of effectiveness measures , 2012, SIGIR '12.

[20]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[21]  W. S. Cooper Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[22]  Madely du Preez Collaborative Information Seeking: The Art and Science of Making the Whole Greater than the Sum of All , 2013, Online Inf. Rev..

[23]  Chirag Shah,et al.  Task, Information Seeking Intentions, and User Behavior: Toward A Multi-level Understanding of Web Search , 2019, CHIIR.

[24]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[25]  Richard L. Oliver,et al.  Satisfaction: A Behavioral Perspective On The Consumer , 1996 .

[26]  Pertti Vakkari,et al.  Evaluation Methodologies in Information Retrieval (Dagstuhl Seminar 13441) , 2013, Dagstuhl Reports.

[27]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[28]  Lois M. L. Delcambre,et al.  Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions , 2008, ECIR.

[29]  James Allan,et al.  Predicting searcher frustration , 2010, SIGIR.

[30]  Ryen W. White Skill discovery in virtual assistants , 2018, Commun. ACM.

[31]  Elaine Toms,et al.  Report on the First Workshop on Supporting Complex Search Tasks , 2015, SIGIR Forum.

[32]  Chirag Shah,et al.  The Broad View of Task Type Using Path Analysis , 2018, ICTIR.

[33]  Chirag Shah,et al.  Bridging Gaps: Predicting User and Task Characteristics from Partial User Information , 2019, SIGIR.

[34]  Chirag Shah,et al.  A Hands-On Introduction to Data Science , 2020 .

[35]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[36]  Pertti Vakkari,et al.  Task complexity, problem structure and information actions - Integrating studies on information seeking and retrieval , 1999, Inf. Process. Manag..

[37]  Ryen W. White,et al.  Characterizing and supporting cross-device search tasks , 2013, WSDM.

[38]  A. Tversky,et al.  Intuitive Prediction: Biases and Corrective Procedures , 1982 .

[39]  Michael Nycyk Review of: Shah, Chirag. Collaborative information seeking: the art and science of making the whole greater than the sum of all. Dordrecht: Springer, 2012 , 2013, Inf. Res..

[40]  Imed Zitouni,et al.  Hey Cortana ! Exploring the use cases of a Desktop based Digital , 2018 .

[41]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[42]  Weng-Keen Wong,et al.  End-user interactions with intelligent and autonomous systems , 2012, CHI Extended Abstracts.

[43]  Paul Thomas,et al.  Measuring the Utility of Search Engine Result Pages: An Information Foraging Based Measure , 2018, SIGIR.

[44]  Pia Borlund,et al.  Experimental components for the evaluation of interactive information retrieval systems , 2000, J. Documentation.

[45]  Nicholas J. Belkin,et al.  Some(what) grand challenges for information retrieval , 2008, SIGF.

[46]  Chirag Shah,et al.  The Paradox of Personalization: Does Task Prediction Require Individualized Models? , 2018, CHIIR.

[47]  Ryen W. White,et al.  Understanding and Predicting Graded Search Satisfaction , 2015, WSDM.

[48]  Mary Czerwinski,et al.  A diary study of task switching and interruptions , 2004, CHI.

[49]  Kalervo Järvelin,et al.  Task Complexity Affects Information Seeking and Use , 1995, Inf. Process. Manag..

[50]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[51]  Ryen W. White,et al.  Modeling and analysis of cross-session search tasks , 2011, SIGIR.

[52]  Emine Yilmaz,et al.  LearnIR: WSDM 2018 Workshop on Learning from User Interactions , 2018, WSDM.

[53]  Nicholas J. Belkin,et al.  Display time as implicit feedback: understanding task effects , 2004, SIGIR '04.

[54]  Chirag Shah,et al.  How Much is Too Much?: Whole Session vs. First Query Behaviors in Task Type Prediction , 2018, SIGIR.

[55]  Reijo Savolainen,et al.  Expectancy-value beliefs and information needs as motivators for task-based information seeking , 2012, J. Documentation.

[56]  Richard David Jacques,et al.  The nature of engagement and its role in hypermedia evaluation and design , 1996 .

[57]  E. Charnov Optimal foraging, the marginal value theorem. , 1976, Theoretical population biology.

[58]  M. Boltz,et al.  The role of learning in remembered duration , 1998, Memory & cognition.

[59]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[60]  Ben Shneiderman,et al.  Creating creativity: user interfaces for supporting innovation , 2000, TCHI.

[61]  Ryen W. White,et al.  Multi-device digital assistance , 2019, Commun. ACM.

[62]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[63]  Eugene Agichtein,et al.  Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior , 2012, WWW.

[64]  Peter Bailey,et al.  Incorporating User Expectations and Behavior into the Measurement of Search Effectiveness , 2017, ACM Trans. Inf. Syst..

[65]  Wei Chu,et al.  Modeling the impact of short- and long-term behavior on search personalization , 2012, SIGIR '12.

[66]  Robert G. Capra,et al.  The Effects of Manipulating Task Determinability on Search Behaviors and Outcomes , 2018, SIGIR.

[67]  Chirag Shah,et al.  Social Information Seeking , 2017, The Information Retrieval Series.

[68]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[69]  Nicholas J. Belkin,et al.  A faceted approach to conceptualizing tasks in information seeking , 2008, Inf. Process. Manag..

[70]  Krisztian Balog Task-completion Engines: A Vision with a Plan , 2015, SCST@ECIR.

[71]  Ryen W. White,et al.  Time-critical search , 2014, SIGIR.

[72]  James Allan,et al.  Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne , 2012, SIGF.

[73]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[74]  Robert G. Capra,et al.  NSF workshop on task-based information search systems , 2013, SIGIR Forum.

[75]  David Mease,et al.  Evaluating web search using task completion time , 2009, SIGIR.

[76]  Nicholas J. Belkin,et al.  Relationships between categories of relevance criteria and stage in task completion , 2007, Inf. Process. Manag..

[77]  Imed Zitouni,et al.  Understanding User Satisfaction with Intelligent Assistants , 2016, CHIIR.

[78]  Ryen W. White,et al.  Struggling and Success in Web Search , 2015, CIKM.

[79]  Ryen W. White,et al.  Characterizing the influence of domain expertise on web search behavior , 2009, WSDM '09.

[80]  Pertti Vakkari,et al.  A theory of the task-based information retrieval process: a summary and generalisation of a longitudinal study , 2001, J. Documentation.

[81]  Ahmed Hassan Awadallah,et al.  Beyond DCG: user behavior as a predictor of a successful search , 2010, WSDM '10.

[82]  C. R. Snyder,et al.  Oxford Handbook of Positive Psychology , 2009 .

[83]  Jane Li,et al.  Good abandonment in mobile and PC internet search , 2009, SIGIR.

[84]  Ryen W. White,et al.  Task Completion Detection: A Study in the Context of Intelligent Systems , 2019, SIGIR.

[85]  Diane Kelly,et al.  Methods for Evaluating Interactive Information Retrieval Systems with Users , 2009, Found. Trends Inf. Retr..

[86]  Robin Jeffries,et al.  Task Behaviors During Web Search: The Difficulty of Assigning Labels , 2009, 2009 42nd Hawaii International Conference on System Sciences.

[87]  Jaime Arguello,et al.  Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework , 2015, ICTIR.

[88]  Ryen W. White Interactions with Search Systems , 2016 .

[89]  Nicholas J. Belkin,et al.  Report on the SIGIR workshop on "entertain me": supporting complex search tasks , 2012, SIGF.

[90]  Nicholas J. Belkin People, Interacting with Information1 , 2016, SIGF.

[91]  Deborah L. McGuinness,et al.  An Intelligent Personal Assistant for Task and Time Management , 2007, AI Mag..

[92]  L A ClarkeCharles,et al.  Report on the SIGIR 2013 workshop on modeling user behavior for information retrieval evaluation (MUBE 2013) , 2013 .

[93]  Mounia Lalmas,et al.  Measuring User Engagement , 2014, Measuring User Engagement.

[94]  D. Allen Getting Things Done: The Art of Stress-Free Productivity , 2001 .

[95]  Diane Kelly Understanding implicit feedback and document preference: a naturalistic user study , 2004, SIGF.

[96]  Ryen W. White,et al.  Intelligent Task Recognition: Towards Enabling Productivity Assistance in Daily Life , 2020, ICMR.

[97]  Chirag Shah,et al.  Collaborative Information Seeking , 2015, Computer Supported Cooperative Work.

[98]  Minyi Guo,et al.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems , 2018, CIKM.

[99]  Nicholas Jing Yuan,et al.  DRN: A Deep Reinforcement Learning Framework for News Recommendation , 2018, WWW.

[100]  R. Block,et al.  The role of attention in time estimation processes , 1996 .

[101]  Christina Lioma,et al.  Report on TBAS 2012: workshop on task-based and aggregated search , 2012, SIGF.