Towards modeling user behavior in human-machine interactions: Effect of Errors and Emotions

Data-driven approaches to spoken dialog strategy design rely on a sound understanding and modeling of user behavior in their interaction with machines. The spoken language usermachine communication channel is inherently noisy; noise in the channel may be due to errors in machine speech recognition, language understanding or other machine/user communication uncertainty and errors. Hence, annotation of human-machine dialogs needs to pay special attention to user behavior under errors and uncertainty. Another key aspect of user behavior is the dynamics of affect or emotions during an interaction relating to "how", contra "what", information is being conveyed. The goal of our work is on developing an account of user behavior working with annotated data from real human-machine mixed initiative dialogs. We illustrate the details of preparing the data for these needs using two case studies. First, we consider the DARPA Communicator dialog corpus to examine categories of error perception, user behavior under error, effect of user strategies on error recovery, and the role of user initiative in error situations. A conditional probability model smoothed by weighted ASR error rate is proposed. Second, we consider data and tagging needs for tracking the “emotional” aspects of human-machine interaction. Toward that end, we used data from a commercially deployed airline information systems and a WoZ study of child-machine interactions. Issues related to capturing emotionally salient information at the lexical and discourse level are highlighted in the context of automatic emotion recognition.