Reinforcement Learning of Cooperative Persuasive Dialogue Policies using Framing

In this paper, we apply reinforcement learning for automatically learning cooperative persuasive dialogue system policies using framing, the use of emotionally charged statements common in persuasive dialogue between humans. In order to apply reinforcement learning, we describe a method to construct user simulators and reward functions specifically tailored to persuasive dialogue based on a corpus of persuasive dialogues between human interlocutors. Then, we evaluate the learned policy and the effect of framing through experiments both with a user simulator and with real users. The experimental evaluation indicates that applying reinforcement learning is effective for construction of cooperative persuasive dialogue systems which use framing.

[1]  Peter A. Heeman,et al.  Representing the Reinforcement Learning state in a negotiation dialogue , 2009, 2009 IEEE Workshop on Automatic Speech Recognition & Understanding.

[2]  Kallirroi Georgila,et al.  Reinforcement Learning of Argumentation Dialogue Policies in Negotiation , 2011, INTERSPEECH.

[3]  David R. Traum,et al.  Prediction of strategy and outcome as negotiation unfolds by using basic verbal and behavioral features , 2013, INTERSPEECH.

[4]  Céline Rouveirol,et al.  Machine Learning: ECML-98 , 1998, Lecture Notes in Computer Science.

[5]  Tomoki Toda,et al.  Dialogue management for leading the conversation in persuasive dialogue systems , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[6]  Allison Sauppé,et al.  A Regression-based Approach to Modeling Addressee Backchannels , 2012, SIGDIAL Conference.

[7]  Ryuichiro Higashinaka,et al.  Wizard of Oz evaluation of listening-oriented dialogue control using POMDP , 2011, 2011 IEEE Workshop on Automatic Speech Recognition & Understanding.

[8]  S. Young,et al.  Scaling POMDPs for Spoken Dialog Management , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[9]  Schneider,et al.  All Frames Are Not Created Equal: A Typology and Critical Analysis of Framing Effects. , 1998, Organizational behavior and human decision processes.

[10]  Valeria Carofiglio,et al.  Portia: A User-Adapted Persuasion System in the Healthy-Eating Domain , 2007, IEEE Intelligent Systems.

[11]  Kallirroi Georgila,et al.  Reinforcement Learning of Question-Answering Dialogue Policies for Virtual Museum Guides , 2012, SIGDIAL Conference.

[12]  Roberto Pieraccini,et al.  A stochastic model of human-machine interaction for learning dialog strategies , 2000, IEEE Trans. Speech Audio Process..

[13]  A. Meystel,et al.  Intelligent Systems , 2001 .

[14]  Kallirroi Georgila.  Reinforcement Learning of Two-Issue Negotiation Dialogue Policies , 2013, SIGDIAL Conference.

[15]  Tomoki Toda,et al.  Construction and Analysis of a Persuasive Dialogue Corpus , 2014, IWSDS.

[16]  Kallirroi Georgila,et al.  Reinforcement Learning of Multi-Issue Negotiation Dialogue Policies , 2015, SIGDIAL Conference.

[17]  Franck Thollard,et al.  Proceedings of COLING , 2004 .

[18]  Ryuichiro Higashinaka,et al.  Controlling Listening-oriented Dialogue using Partially Observable Markov Decision Processes , 2010, COLING.

[19]  Judith Masthoff,et al.  Persuasive Effects of Embodied Conversational Agent Teams , 2007, HCI.

[20]  Martin A. Riedmiller.  Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method , 2005, ECML.

[21]  Roie Zivan,et al.  POMDP based Negotiation Modeling , 2009 .

[22]  Fiorella de Rosis,et al.  Artifices for Persuading to Improve Eating Habits , 2006, AAAI Spring Symposium: Argumentation for Consumers of Healthcare.