Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns

Text-based conversational systems, also referred to as chatbots, have grown widely popular. Current natural language understanding technologies are not yet ready to tackle the complexities in conversational interactions. Breakdowns are common, leading to negative user experiences. Guided by communication theories, we explore user preferences for eight repair strategies, including ones that are common in commercially-deployed chatbots (e.g., confirmation, providing options), as well as novel strategies that explain characteristics of the underlying machine learning algorithms. We conducted a scenario-based study to compare repair strategies with Mechanical Turk workers (N=203). We found that providing options and explanations were generally favored, as they manifest initiative from the chatbot and are actionable to recover from breakdowns. Through detailed analysis of participants' responses, we provide a nuanced understanding on the strengths and weaknesses of each repair strategy.

[1]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[2]  H. A. David,et al.  The method of paired comparisons , 1966 .

[3]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[4]  E. Schegloff,et al.  The preference for self-correction in the organization of repair in conversation , 1977 .

[5]  Nancy Larson-Powers,et al.  PAIRED COMPARISON AND TIME-INTENSITY MEASUREMENTS OF THE SENSORY PROPERTIES OF BEVERAGES AND GELATINS CONTAINING SUCROSE OR SYNTHETIC SWEETENERS , 1978 .

[6]  B. Brinton,et al.  Development of conversational repair strategies in response to requests for clarification. , 1986, Journal of speech and hearing research.

[7]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[8]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[9]  A. Agresti.  Categorical data analysis , 1993 .

[10]  Ben Shneiderman,et al.  Direct manipulation vs. interface agents , 1997, INTR.

[11]  Eric Horvitz,et al.  Uncertainty, Utility, and Misunderstanding: A Decision-Theoretic Perspective on Grounding in Conversational Systems , 1999 .

[12]  Janet E. Cahn,et al.  A Psychological Model of Grounding and Repair in Dialog , 1999 .

[13]  David Traum,et al.  Computational Models of Grounding in Collaborative Systems , 1999 .

[14]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[15]  Eric Horvitz,et al.  Grounding Criterion: Toward a Formal Theory of Grounding , 2000 .

[16]  Susan E. Brennan,et al.  The Grounding Problem in Conversations With and Through Computers , 2000 .

[17]  Duncan Cramer,et al.  The Sage dictionary of statistics : a practical resource for students in the social sciences , 2004 .

[18]  H. Chad Lane,et al.  Design recommendations to support automated explanation and tutoring , 2005 .

[19]  Vanda Broughton,et al.  Sage Dictionary of Statistics: A Practical Resource for Students in the Social Sciences , 2005 .

[20]  Lee Lacy,et al.  Defense Advanced Research Projects Agency (DARPA) Agent Markup Language Computer Aided Knowledge Acquisition , 2005 .

[21]  Thomas G. Dietterich,et al.  Toward harnessing user feedback for machine learning , 2007, IUI '07.

[22]  Sylvain Choisel,et al.  Evaluation of multichannel reproduced sound: scaling auditory attributes underlying listener preference. , 2007, The Journal of the Acoustical Society of America.

[23]  Helvi Kyngäs,et al.  The qualitative content analysis process. , 2008, Journal of advanced nursing.

[24]  Chin-Laung Lei,et al.  A crowdsourceable QoE evaluation framework for multimedia content , 2009, ACM Multimedia.

[25]  Y. Wilks,et al.  Book Review: Close Engagements with Artificial Companions: Key Social, Psychological, Ethical, and Design Issues edited by Yorick Wilks , 2010, CL.

[26]  Jodi Forlizzi,et al.  Receptionist or information kiosk: how do people talk with a robot? , 2010, CSCW '10.

[27]  Siddhartha S. Srinivasa,et al.  Gracefully mitigating breakdowns in robotic services , 2010, HRI 2010.

[28]  David Firth,et al.  Bradley-Terry Models in R: The BradleyTerry2 Package , 2012 .

[29]  M. Napierala.  What Is the Bonferroni Correction ? , 2014 .

[30]  Geoffrey Zweig,et al.  Rapidly Scaling Dialog Systems with Interactive Learning , 2015, IWSDS.

[31]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[32]  Florence March,et al.  2016 , 2016, Affair of the Heart.

[33]  Abigail Sellen,et al.  "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents , 2016, CHI.

[34]  Leila Takayama,et al.  Help Me Please: Robot Politeness Strategies for Soliciting Help From Humans , 2016, CHI.

[35]  Alexander I. Rudnicky,et al.  A Wizard-of-Oz Study on A Non-Task-Oriented Dialog Systems That Reacts to User Engagement , 2016, SIGDIAL Conference.

[36]  N. Sadat Shami,et al.  What Can You Do?: Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees , 2016, Conference on Designing Interactive Systems.

[37]  Marcos Serrano,et al.  Visual Composition of Graphical Elements on Non-Rectangular Displays , 2017, CHI.

[38]  Iolanda Leite,et al.  Better Faulty than Sorry: Investigating Social Recovery Strategies to Minimize the Impact of Failure in Human-Robot Interaction , 2017, WCIHAI@IVA.

[39]  Jacki O'Neill,et al.  How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians , 2017, INTERACT.

[40]  Anne Roudaut,et al.  Frozen Suit: Designing a Changeable Stiffness Suit and its Application to Haptic Games , 2017, CHI.

[41]  Shwetak N. Patel,et al.  FarmChat: A Conversational Agent to Answer Farmer Queries , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[42]  Shwetak N. Patel,et al.  Evaluating and Informing the Design of Chatbots , 2018, Conference on Designing Interactive Systems.

[43]  Sarah Sharples,et al.  Voice Interfaces in Everyday Life , 2018, CHI.

[44]  Jichen Zhu,et al.  Patterns for How Users Overcome Obstacles in Voice User Interfaces , 2018, CHI.

[45]  Shwetak N. Patel,et al.  FarmChat , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[46]  Shwetak N. Patel,et al.  Convey: Exploring the Use of a Context View for Chatbots , 2018, CHI.

[47]  Yasaman Khazaeni,et al.  All Work and No Play? Conversations with a Question-and-Answer Chatbot in the Wild , 2018, CHI 2018.

[48]  Yasaman Khazaeni,et al.  All Work and No Play? , 2018, CHI.

[49]  Justin D. Weisz,et al.  BigBlueBot: teaching strategies for successful human-agent interactions , 2019, IUI.

[50]  BigBlueBot , 2019, Proceedings of the 24th International Conference on Intelligent User Interfaces.