Reinforcement Learning of Question-Answering Dialogue Policies for Virtual Museum Guides

We use Reinforcement Learning (RL) to learn question-answering dialogue policies for a real-world application. We analyze a corpus of interactions of museum visitors with two virtual characters that serve as guides at the Museum of Science in Boston, in order to build a realistic model of user behavior when interacting with these characters. A simulated user is built based on this model and used for learning the dialogue policy of the virtual characters using RL. Our learned policy outperforms two baselines (including the original dialogue policy that was used for collecting the corpus) in a simulation setting.

[1]  Steve J. Young,et al.  Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems , 2010, Comput. Speech Lang..

[2]  Athanasios Katsamanis,et al.  The Twins Corpus of Museum Visitor Questions , 2012, LREC.

[3]  Ron Artstein,et al.  An Integrated Authoring Tool for Tactical Questioning Dialogue Systems , 2009 .

[4]  Anton Leuski,et al.  Practical Language Processing for Virtual Humans , 2010, IAAI.

[5]  Michelle L. Gregory,et al.  From Question Answering to Visual Exploration , 2006, SIGIR 2006.

[6]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[7]  Steve J. Young,et al.  Reinforcement learning for parameter estimation in statistical spoken dialogue systems , 2012, Comput. Speech Lang..

[8]  Harry Bunt,et al.  From question answering to spoken dialogue: towards an information search assistant for interactive multimodal information extraction , 2005, INTERSPEECH.

[9]  Joel R. Tetreault,et al.  A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems , 2008, Speech Commun..

[10]  Kallirroi Georgila,et al.  Learning Culture-Specific Dialogue Models from Non Culture-Specific Data , 2011, HCI.

[11]  Kurt VanLehn,et al.  Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies , 2011, User Modeling and User-Adapted Interaction.

[12]  Anton Leuski,et al.  Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides , 2010, IVA.

[13]  Oliver Lemon,et al.  Does this list contain what you were searching for? Learning adaptive dialogue strategies for interactive question answering , 2009, Natural Language Engineering.

[14]  Hua Ai,et al.  Assessing Dialog System User Simulation Evaluation Measures Using Human Judges , 2008, ACL.

[15]  Sebastian Varges,et al.  Interactive Question Answering and Constraint Relaxation in Spoken Dialogue Systems , 2006, Natural Language Engineering.

[16]  Kallirroi Georgila,et al.  Learning Dialogue Strategies from Older and Younger Simulated Users , 2010, SIGDIAL Conference.

[17]  Anton Leuski,et al.  Building Effective Question Answering Characters , 2006, SIGDIAL Workshop.

[18]  Kallirroi Georgila,et al.  User simulation for spoken dialogue systems: learning and evaluation , 2006, INTERSPEECH.

[19]  Sebastian Varges,et al.  Interactive Question Answering and Constraint Relaxation in Spoken Dialogue Systems , 2006, SIGDIAL Workshop.

[20]  Lihong Li,et al.  Reinforcement learning for dialog management using least-squares Policy iteration and fast feature selection , 2009, INTERSPEECH.

[21]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track , 2001, LREC.

[22]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[23]  Satoshi Nakamura,et al.  Modeling Spoken Decision Making Dialogue and Optimization of its Dialogue Strategy , 2010, SIGDIAL Conference.

[24]  Peter A. Heeman,et al.  Representing the Reinforcement Learning state in a negotiation dialogue , 2009, 2009 IEEE Workshop on Automatic Speech Recognition & Understanding.

[25]  Kallirroi Georgila,et al.  Reinforcement Learning of Argumentation Dialogue Policies in Negotiation , 2011, INTERSPEECH.

[26]  Arne Jönsson,et al.  Experiences from Combining Dialogue System Development with Information Extraction Techniques , 2004, New Directions in Question Answering.