Optimising Incremental Dialogue Decisions Using Information Density for Interactive Systems

Incremental processing allows system designers to address several discourse phenomena that have previously been somewhat neglected in interactive systems, such as backchannels or barge-ins, but that can enhance the responsiveness and naturalness of systems. Unfortunately, prior work has focused largely on deterministic incremental decision making, rendering system behaviour less flexible and adaptive than is desirable. We present a novel approach to incremental decision making that is based on Hierarchical Reinforcement Learning to achieve an interactive optimisation of Information Presentation (IP) strategies, allowing the system to generate and comprehend backchannels and barge-ins, by employing the recent psycholinguistic hypothesis of information density (ID) (Jaeger, 2010). Results in terms of average rewards and a human rating study show that our learnt strategy outperforms several baselines that are not sensitive to ID by more than 23%.

[1]  Kallirroi Georgila,et al.  Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets , 2008, CL.

[2]  Claude E. Shannon,et al.  A Mathematical Theory of Communications , 1948 .

[3]  David Schlangen,et al.  Evaluation and Optimisation of Incremental Processors , 2011, Dialogue Discourse.

[4]  Oliver Lemon,et al.  Adaptive Information Presentation for Spoken Dialogue Systems : Evaluation with human subjects , 2011 .

[5]  Alice Turk,et al.  The Smooth Signal Redundancy Hypothesis: A Functional Explanation for Relationships between Redundancy, Prosodic Prominence, and Duration in Spontaneous Speech , 2004, Language and speech.

[6]  Roger Levy,et al.  Speakers optimize information density through syntactic reduction , 2006, NIPS.

[7]  Jason D. Williams,et al.  Stability and Accuracy in Incremental Speech Recognition , 2011, SIGDIAL Conference.

[8]  Maxine Eskénazi,et al.  A Finite-State Turn-Taking Model for Spoken Dialog Systems , 2009, NAACL.

[9]  Nina Dethlefs,et al.  Spatially-aware dialogue control using hierarchical reinforcement learning , 2011, TSLP.

[10]  Nina Dethlefs,et al.  Combining Hierarchical Reinforcement Learning and Bayesian Networks for Natural Language Generation in Situated Dialogue , 2011, ENLG.

[11]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[12]  Blaise Roger Marie Thomson,et al.  Statistical methods for spoken dialogue management , 2013 .

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Thierry Dutoit,et al.  A probabilistic framework for dialog simulation and optimal strategy learning , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[15]  S. Singh,et al.  Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System , 2011, J. Artif. Intell. Res..

[16]  Thomas G. Dietterich.  Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[17]  Stephen Young.  Probabilistic methods in spoken–dialogue systems , 2000, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[18]  Roberto Pieraccini,et al.  A stochastic model of human-machine interaction for learning dialog strategies , 2000, IEEE Trans. Speech Audio Process..

[19]  Menno van Zaanen.  Bootstrapping Syntax and Recursion using Alginment-Based Learning , 2000, ICML.

[20]  Oliver Lemon,et al.  Optimising Information Presentation for Spoken Dialogue Systems , 2010, ACL.

[21]  Michael White,et al.  Linguistically Motivated Complementizer Choice in Surface Realization , 2011 .

[22]  Oliver Lemon,et al.  Evaluation of a hierarchical reinforcement learning spoken dialogue system , 2010, Comput. Speech Lang..

[23]  Gabriel Skantze,et al.  Towards Incremental Speech Generation in Dialogue Systems , 2010, SIGDIAL Conference.

[24]  Dan Jurafsky,et al.  Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. , 2003, The Journal of the Acoustical Society of America.

[25]  Heriberto Cuayáhuitl,et al.  Hierarchical Reinforcement Learning for Spoken Dialogue Systems , 2009 .

[26]  Marilyn A. Walker,et al.  An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email , 2000, J. Artif. Intell. Res..

[27]  Oliver Lemon,et al.  Learning to Adapt to Unknown Users: Referring Expression Generation in Spoken Dialogue Systems , 2010, ACL.

[28]  Milica Gasic,et al.  The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management , 2010, Comput. Speech Lang..

[29]  Gabriel Skantze,et al.  A General, Abstract Model of Incremental Dialogue Processing , 2009, EACL.

[30]  Matthew Purver,et al.  Incremental Generation by Incremental Parsing , 2003 .

[31]  T. Florian Jaeger,et al.  Redundancy and reduction: Speakers manage syntactic information density , 2010, Cognitive Psychology.

[32]  Oliver Lemon,et al.  Adaptive Information Presentation for Spoken Dialogue Systems: Evaluation with real users , 2011, ENLG.

[33]  Wolfgang Finkler,et al.  Incremental generation for real-time applications , 1995 .

[34]  David DeVault,et al.  Can I Finish? Learning When to Respond to Incremental Interpretation Results in Interactive Dialogue , 2009, SIGDIAL Conference.

[35]  Oliver Lemon,et al.  Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation , 2011, Comput. Speech Lang..

[36]  Eugene Charniak,et al.  Entropy Rate Constancy in Text , 2002, ACL.

[37]  Roberto Pieraccini,et al.  A stochastic model of computer-human interaction for learning dialogue strategies , 1997, EUROSPEECH.

[38]  David Schlangen,et al.  Collaborating on Utterances with a Spoken Dialogue System Using an ISU-based Approach to Incremental Dialogue Management , 2010, SIGDIAL Conference.

[39]  Helen F. Hastie,et al.  Optimising Incremental Generation for Spoken Dialogue Systems: Reducing the Need for Fillers , 2012, INLG.