PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism

We investigate response generation for multi-turn dialogue in generative chatbots. Existing generative modelsbased on RNNs (Recurrent Neural Networks) usually employ the last hidden state to summarize the history, which makesmodels unable to capture the subtle variability observed in different dialogues and cannot distinguish the differencesbetween dialogues that are similar in composition. In this paper, we propose Pseudo-Variational Gated Recurrent Unit (PVGRU). The key novelty of PVGRU is a recurrent summarizing variable thataggregates the accumulated distribution variations of subsequences. We train PVGRU without relying on posterior knowledge, thus avoiding the training-inference inconsistency problem. PVGRU can perceive subtle semantic variability through summarizing variables that are optimized by two objectives we employ for training: distribution consistency and reconstruction. In addition, we build a Pseudo-Variational Hierarchical Dialogue(PVHD) model based on PVGRU. Experimental results demonstrate that PVGRU can broadly improve the diversity andrelevance of responses on two benchmark datasets.

[1]  Hinrich Schütze,et al.  Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages , 2022, ACL.

[2]  Daling Wang,et al.  MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generation , 2022, COLING.

[3]  Yi Mao,et al.  DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation , 2022, Annual Meeting of the Association for Computational Linguistics.

[4]  Yan Wang,et al.  BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data , 2021, ACL.

[5]  Shaoxiong Feng,et al.  Generating Relevant and Coherent Dialogue Responses using Self-Separated Conditional Variational AutoEncoders , 2021, ACL.

[6]  Xuelong Li,et al.  Attention-Emotion-Enhanced Convolutional LSTM for Sentiment Analysis , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Peng Xu,et al.  Variational Transformers for Diverse Response Generation , 2020, ArXiv.

[8]  Lili Mou,et al.  Adversarial Learning on the Latent Space for Diverse Dialog Generation , 2019, COLING.

[9]  Mohammad Norouzi,et al.  Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse , 2019, NeurIPS.

[10]  Jianfeng Gao,et al.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation , 2019, ACL.

[11]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[12]  Hua Wu,et al.  PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable , 2019, ACL.

[13]  Chris Callison-Burch,et al.  ChatEval: A Tool for Chatbot Evaluation , 2019, NAACL.

[14]  Anoop Cherian,et al.  Audio Visual Scene-Aware Dialog , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Verena Rieser,et al.  Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity , 2018, EMNLP.

[16]  Xu Sun,et al.  An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation , 2018, EMNLP.

[17]  Weinan Zhang,et al.  Context-Sensitive Generation of Open-Domain Conversational Responses , 2018, COLING.

[18]  Xiaoyan Zhu,et al.  Generating Informative Responses with Controlled Sentence Function , 2018, ACL.

[19]  Kai Liu,et al.  Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task , 2018, QA@ACL.

[20]  Xiaodong Gu,et al.  DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder , 2018, ICLR.

[21]  Joachim Denzler,et al.  Detecting Regions of Maximal Divergence for Spatio-Temporal Anomaly Detection , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Gunhee Kim,et al.  A Hierarchical Latent Structure for Variational Conversation Modeling , 2018, NAACL.

[23]  Maxine Eskénazi,et al.  Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation , 2018, ACL.

[24]  Pascal Poupart,et al.  Variational Attention for Sequence-to-Sequence Models , 2017, COLING.

[25]  Pascal Poupart,et al.  Why Do Neural Dialog Systems Generate Short and Meaningless Replies? a Comparison between Dialog and Translation , 2017, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Xiaoyu Shen,et al.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset , 2017, IJCNLP.

[27]  Maxine Eskénazi,et al.  Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders , 2017, ACL.

[28]  Wei-Ying Ma,et al.  Hierarchical Recurrent Attention Network for Response Generation , 2017, AAAI.

[29]  Alan Ritter,et al.  Adversarial Learning for Neural Dialogue Generation , 2017, EMNLP.

[30]  Joelle Pineau,et al.  A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues , 2016, AAAI.

[31]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[32]  Jianfeng Gao,et al.  A Diversity-Promoting Objective Function for Neural Conversation Models , 2015, NAACL.

[33]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[34]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[35]  Hang Li,et al.  Neural Responding Machine for Short-Text Conversation , 2015, ACL.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[38]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[39]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[40]  Colin Cherry,et al.  A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU , 2014, WMT@ACL.

[41]  S. Williams,et al.  Pearson's correlation coefficient. , 1996, The New Zealand medical journal.

[42]  Lei Li,et al.  Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation , 2020, ICML.

[43]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[44]  Xu Sun,et al.  Diversity-Promoting GAN: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation , 2018, EMNLP.

[45]  Xuan Wang,et al.  Variational Autoregressive Decoder for Neural Response Generation , 2018, EMNLP.

[46]  Matthew Henderson,et al.  Word-Based Dialog State Tracking with Recurrent Neural Networks , 2014, SIGDIAL Conference.