Evaluating Spoken Dialogue Processing for Time-Offset Interaction

This paper presents the first evaluation of a full automated prototype system for time-offset interaction, that is, conversation between a live person and recordings of someone who is not temporally copresent. Speech recognition reaches word error rates as low as 5% with generalpurpose language models and 19% with domain-specific models, and language understanding can identify appropriate direct responses to 60‐66% of user utterances while keeping errors to 10‐16% (the remainder being indirect, or off-topic responses). This is sufficient to enable a natural flow and relatively open-ended conversations, with a collection of under 2000 recorded statements.

[1]  Alexander I. Rudnicky,et al.  Sorry and I Didn’t Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies , 2005, SIGDIAL.

[2]  Kallirroi Georgila,et al.  New Dimensions in Testimony Demonstration , 2016, NAACL.

[3]  Anton Leuski,et al.  How Many Utterances Are Needed to Support Time-Offset Interaction? , 2015, FLAIRS Conference.

[4]  Ron Artstein.  Error Return Plots , 2011, SIGDIAL Conference.

[5]  Kallirroi Georgila,et al.  Time-offset interaction with a holocaust survivor , 2014, IUI.

[6]  PauschRandy,et al.  Carnegie Mellon's entertainment technology center , 2007 .

[7]  Scott Stevens,et al.  Synthetic interviews: the art of creating a “dyad” between humans and machine-based characters , 1998, MULTIMEDIA '98.

[8]  Ronald Rosenfeld,et al.  Statistical language modeling using the CMU-cambridge toolkit , 1997, EUROSPEECH.

[9]  David R. Traum,et al.  A reranking approach for recognition and classification of speech input in conversational dialogue systems , 2012, 2012 IEEE Spoken Language Technology Workshop (SLT).

[10]  David R. Traum,et al.  I've said it before, and I'll say it again: An empirical investigation of the upper bound of the selection approach to dialogue , 2010, SIGDIAL Conference.

[11]  Anton Leuski,et al.  NPCEditor: Creating Virtual Human Dialogue Using Information Retrieval Techniques , 2011, AI Mag..

[12]  Anton Leuski,et al.  Building Effective Question Answering Characters , 2006, SIGDIAL Workshop.

[13]  Anton Leuski,et al.  Semi-formal Evaluation of Conversational Characters , 2009, Languages: From Formal to Natural.