Exploring Conversational Search With Humans, Assistants, and Wizards

Chatbots and conversational assistants are becoming increasingly popular. However, for information seeking scenarios, these systems still have very limited conversational abilities, and primarily serve as proxies to existing web search engines. In this work, we ask: what would conversational search look like with a truly intelligent assistant? To begin answering this question empirically, we conduct a user study, in which 21 participants are each given 3 information seeking tasks to solve using a text-based chat interface. To complete each task, participants conversed with three conversational agents: an existing commercial system, a human expert, and a perceived experimental automatic system, backed by a human 'wizard' behind the curtain. The observations and insights of our study help us understand the aspirations of users and the limitations of the current conversational agents -- and to sharpen a frontier of work required to improve conversational assistants for search scenarios.

[1]  Wen-tau Yih,et al.  Web-based Question Answering: Revisiting AskMSR , 2015 .

[2]  Eugene Agichtein,et al.  When web search fails, searchers become askers: understanding the transition , 2012, SIGIR '12.

[3]  Ben Carterette,et al.  Overview of the TREC 2014 Session Track , 2014, TREC.

[4]  Filippo Menczer,et al.  The rise of social bots , 2014, Commun. ACM.

[5]  Ravi Kumar,et al.  Search in the Lost Sense of "Query": Question Formulation in Web Search Queries and its Temporal Changes , 2011, ACL.

[6]  Patric R. Spence,et al.  Is that a bot running the social media feed? Testing the differences in perceptions of communication quality for a human agent and a bot agent on Twitter , 2014, Comput. Hum. Behav..

[7]  Marti A. Hearst.  'Natural' search user interfaces , 2011, CACM.

[8]  Filip Radlinski,et al.  A Theoretical Framework for Conversational Search , 2017, CHIIR.

[9]  E. Aharoni,et al.  Social reactions toward people vs. computers: How mere lables shape interactions , 2007, Comput. Hum. Behav..

[10]  Eugene Agichtein,et al.  What Do You Mean Exactly?: Analyzing Clarification Questions in CQA , 2017, CHIIR.

[11]  Anne Aula,et al.  How does search behavior change as search becomes more difficult? , 2010, CHI.

[12]  Matthieu J. Guitton,et al.  Interacting with bots online: Users' reactions to actions of automated programs in Wikipedia , 2015, Comput. Hum. Behav..

[13]  Ryen W. White,et al.  Questions vs. Queries in Informational Search Tasks , 2015, WWW.

[14]  Serge Demeyer,et al.  Among the Machines: Human-Bot Interaction on Social Q&A Websites , 2016, CHI Extended Abstracts.

[15]  Abigail Sellen,et al.  "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents , 2016, CHI.