Entrainable Neural Conversation Model Based on Reinforcement Learning

The synchronization of words in conversation, called entrainment, is generally observed in human-human conversations. Entrainment has a high correlation with dialogue success, naturalness, and engagement. In this article, we define entrainment scores based on the word similarities in semantic space to evaluate the entrainment of system generation. We optimized a neural conversation model to the entrainment scores using reinforcement learning so that the system can control the degree of entrainment of the system response. Experimental results showed that the proposed entrainable neural conversation model generated comparable or more natural responses than conventional models and satisfactorily controlled the degree of entrainment of the generated responses.

[1]  Christopher Joseph Pal,et al.  Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study , 2019, ACL.

[2]  Rivka Levitan,et al.  Entrainment in Spoken Dialogue Systems: Adopting, Predicting and Influencing User Behavior , 2013, NAACL.

[3]  Marc'Aurelio Ranzato,et al.  Sequence Level Training with Recurrent Neural Networks , 2015, ICLR.

[4]  Julia Hirschberg,et al.  Entrainment, dominance and alliance in supreme court hearings , 2014, Knowl. Based Syst..

[5]  Wei-Ying Ma,et al.  Hierarchical Recurrent Attention Network for Response Generation , 2017, AAAI.

[6]  H. H. Clark,et al.  Conceptual pacts and lexical choice in conversation. , 1996, Journal of experimental psychology. Learning, memory, and cognition.

[7]  J. Pennebaker,et al.  Linguistic Style Matching in Social Interaction , 2002 .

[8]  Hang Li,et al.  “ Tony ” DNN Embedding for “ Tony ” Selective Read for “ Tony ” ( a ) Attention-based Encoder-Decoder ( RNNSearch ) ( c ) State Update s 4 SourceVocabulary Softmax Prob , 2016 .

[9]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[10]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[11]  F. Drasgow,et al.  The polyserial correlation coefficient , 1982 .

[12]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[13]  Arthur Ward.  Measuring Convergence and Priming in Tutorial Dialog , 2007 .

[14]  Leonidas J. Guibas,et al.  A metric for distributions with applications to image databases , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[15]  Maxine Eskénazi,et al.  Prosodic Entrainment in an Information-Driven Dialog System , 2012, INTERSPEECH.

[16]  Martin J. Pickering,et al.  Alignment as the Basis for Successful Communication , 2006 .

[17]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[18]  Lantao Yu,et al.  SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient , 2016, AAAI.

[19]  Dongyan Zhao,et al.  How to Make Context More Useful? An Empirical Study on Context-Aware Neural Conversational Models , 2017, ACL.

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Satoshi Nakamura,et al.  Analyzing the Effect of Entrainment on Dialogue Acts , 2016, SIGDIAL Conference.

[22]  Jiaheng Lu,et al.  Neural Conversation Generation with Auxiliary Emotional Supervised Models , 2019, ACM Trans. Asian Low Resour. Lang. Inf. Process..

[23]  Osmar R. Zaïane,et al.  Automatic Dialogue Generation with Expressed Emotions , 2018, NAACL.

[24]  Alan Ritter,et al.  Adversarial Learning for Neural Dialogue Generation , 2017, EMNLP.

[25]  Jianfeng Gao,et al.  A Persona-Based Neural Conversation Model , 2016, ACL.

[26]  Panayiotis G. Georgiou,et al.  Modeling Interpersonal Linguistic Coordination in Conversations using Word Mover's Distance , 2019, INTERSPEECH.

[27]  Zhe Gan,et al.  Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization , 2018, NeurIPS.

[28]  Michael Werman,et al.  Fast and robust Earth Mover's Distances , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[29]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[30]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[31]  Johanna D. Moore,et al.  Predicting Success in Dialogue , 2007, ACL.

[32]  Matt J. Kusner,et al.  From Word Embeddings To Document Distances , 2015, ICML.

[33]  Benjamin Weiss,et al.  Talker Quality in Interactive Scenarios , 2019, Talker Quality in Human and Machine Interaction.

[34]  M. Natale.  CONVERGENCE OF MEAN VOCAL INTENSITY IN DYADIC COMMUNICATION AS A FUNCTION OF SOCIAL DESIRABILITY , 1975 .

[35]  H. Giles,et al.  Speech Accommodation Theory: The First Decade and Beyond , 1987 .

[36]  Wei-Ying Ma,et al.  Topic Aware Neural Response Generation , 2016, AAAI.

[37]  Satoshi Nakamura,et al.  Neural Conversation Model Controllable by Given Dialogue Act Based on Adversarial Learning and Label-aware Objective , 2019, INLG.

[38]  Guangyou Zhou,et al.  Topic-enhanced emotional conversation generation with attention mechanism , 2019, Knowl. Based Syst..

[39]  Nick Campbell,et al.  Comparing measures of synchrony and alignment in dialogue speech timing with respect to turn-taking activity , 2010, INTERSPEECH.

[40]  Quoc V. Le,et al.  A Neural Conversational Model , 2015, ArXiv.

[41]  Julia Hirschberg,et al.  High Frequency Word Entrainment in Spoken Dialogue , 2008, ACL.

[42]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[43]  Michael Werman,et al.  A Linear Time Histogram Metric for Improved SIFT Matching , 2008, ECCV.

[44]  Julia Hirschberg,et al.  Entrainment and Turn-Taking in Human-Human Dialogue , 2015, AAAI Spring Symposia.