Evaluating State Representations for Reinforcement Learning of Turn-Taking Policies in Tutorial Dialogue

Learning and improving natural turn-taking behaviors for dialogue systems is a topic of growing importance. In task-oriented dialogue where the user can engage in task actions in parallel with dialogue, unrestricted turn taking may be particularly important for dialogue success. This paper presents a novel Markov Decision Process (MDP) representation of dialogue with unrestricted turn taking and a parallel task stream in order to automatically learn effective turn-taking policies for a tutorial dialogue system from a corpus. It also presents and evaluates an approach to automatically selecting features for an MDP state representation of this dialogue. The results suggest that the MDP formulation and the feature selection framework hold promise for learning effective turn-taking policies in taskoriented dialogue systems.

[1]  V. Shute.  Focus on Formative Feedback , 2007 .

[2]  Hideki Kashioka,et al.  Simultaneous feature selection and parameter optimization for training of dialog policy by reinforcement learning , 2012, 2012 IEEE Spoken Language Technology Workshop (SLT).

[3]  Diane J. Litman,et al.  Adapting to Multiple Affective States in Spoken Dialogue , 2012, SIGDIAL Conference.

[4]  Lihong Li,et al.  Reinforcement learning for dialog management using least-squares Policy iteration and fast feature selection , 2009, INTERSPEECH.

[5]  Andrew Olney,et al.  Mining Collaborative Patterns in Tutorial Dialogues , 2010, EDM 2010.

[6]  Joel R. Tetreault,et al.  A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems , 2008, Speech Commun..

[7]  Maxine Eskénazi,et al.  Optimizing the turn-taking behavior of task-oriented spoken dialog systems , 2012, TSLP.

[8]  Kallirroi Georgila,et al.  Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets , 2008, CL.

[9]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[10]  S. Singh,et al.  Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System , 2011, J. Artif. Intell. Res..

[11]  Jason D. Williams,et al.  Integrating Incremental Speech Recognition and POMDP-Based Dialogue Systems , 2012, SIGDIAL Conference.

[12]  Kurt VanLehn,et al.  An Evaluation of Pedagogical Tutorial Tactics for a Natural Language Tutoring System: A Reinforcement Learning Approach , 2011, Int. J. Artif. Intell. Educ..

[13]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[14]  Michael English,et al.  Learning Mixed Initiative Dialog Strategies By Using Reinforcement Learning On Both Conversants , 2005, HLT.