Chinese Emotional Dialogue Response Generation via Reinforcement Learning

In an open-domain dialogue system, recognition and expression of emotions are the key factors for success. Most of the existing research related to Chinese dialogue systems aims at improving the quality of content but ignores the expression of human emotions. In this article, we propose a Chinese emotional dialogue response generation algorithm based on reinforcement learning that can generate responses not only according to content but also according to emotion. In the proposed method, a multi-emotion classification model is first used to add emotion labels to the corpus of post-response pairs. Then, with the help of reinforcement learning, the reward function is constructed based on two aspects, namely, emotion and content. Among the generated candidates, the system selects the one with long-term success as the best reply. At the same time, to avoid safe responses and diversify dialogue, a diversity beam search algorithm is applied in the decoding process. The comparative experiments demonstrate that the proposed model achieves satisfactory results according to both automatic and human evaluations.

[1]  Ashwin K. Vijayakumar,et al.  Diverse Beam Search for Improved Description of Complex Scenes , 2018, AAAI.

[2]  Qinghua Hu,et al.  Combining heterogeneous deep neural networks with conditional random fields for Chinese dialogue act recognition , 2015, Neurocomputing.

[3]  Eduard H. Hovy,et al.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF , 2016, ACL.

[4]  Hao Gao,et al.  Hybrid Feature Selection Algorithm Based on Discrete Artificial Bee Colony for Parkinson Diagnosis , 2021, ACM Trans. Internet Techn..

[5]  Wei-Ying Ma,et al.  Topic Aware Neural Response Generation , 2016, AAAI.

[6]  Zhenbing Liu,et al.  MADNet: A Fast and Lightweight Network for Single-Image Super Resolution , 2020, IEEE Transactions on Cybernetics.

[7]  Zhenbing Liu,et al.  Cascading and Enhanced Residual Networks for Accurate Single-Image Super-Resolution , 2020, IEEE Transactions on Cybernetics.

[8]  Xiang Li,et al.  StalemateBreaker: A Proactive Content-Introducing Approach to Automatic Human-Computer Conversation , 2016, IJCAI.

[9]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[10]  Alan Ritter,et al.  Data-Driven Response Generation in Social Media , 2011, EMNLP.

[11]  Xiao Sun,et al.  Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features , 2016, Neurocomputing.

[12]  Douglas H. Norrie,et al.  Schema-based conversation modeling for agent-oriented manufacturing systems , 2001, Comput. Ind..

[13]  Zhi Zhang,et al.  Research on text sentiment analysis based on CNNs and SVM , 2018, 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA).

[14]  Susan McRoy,et al.  YAG: A Template-Based Text Realization System for Dialog , 2001, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[15]  Alexander M. Rush,et al.  Sequence-to-Sequence Learning as Beam-Search Optimization , 2016, EMNLP.

[16]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[17]  Yuxi Li,et al.  Deep Reinforcement Learning , 2018, Reinforcement Learning for Cyber-Physical Systems.

[18]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[19]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[20]  Jianfeng Gao,et al.  Deep Reinforcement Learning for Dialogue Generation , 2016, EMNLP.

[21]  Bowen Zhou,et al.  Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation , 2016, AAAI.

[22]  Markus Freitag,et al.  Beam Search Strategies for Neural Machine Translation , 2017, NMT@ACL.

[23]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[24]  Jing Li,et al.  A Manually Annotated Chinese Corpus for Non-task-oriented Dialogue Systems , 2018, ArXiv.

[25]  Yingshu Li,et al.  A worker-selection incentive mechanism for optimizing platform-centric mobile crowdsourcing systems , 2020, Comput. Networks.

[26]  Jiguo Yu,et al.  A Differential-Private Framework for Urban Traffic Flows Estimation via Taxi Companies , 2019, IEEE Transactions on Industrial Informatics.

[27]  Geoffrey Zweig,et al.  Joint semantic utterance classification and slot filling with recursive neural networks , 2014, 2014 IEEE Spoken Language Technology Workshop (SLT).

[28]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[29]  Gary Geunbae Lee,et al.  Example-based dialog modeling for practical multi-domain dialog system , 2009, Speech Commun..

[30]  Dongyan Zhao,et al.  RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems , 2017, AAAI.

[31]  John C. S. Lui,et al.  A Reinforcement Learning Approach to Optimize Discount and Reputation Tradeoffs in E-commerce Systems , 2020, ACM Trans. Internet Techn..

[32]  Daniel Jurafsky,et al.  A Simple, Fast Diverse Decoding Algorithm for Neural Generation , 2016, ArXiv.

[33]  Jing He,et al.  A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems , 2016, INTERSPEECH.

[34]  Yicong Zhou,et al.  Prior Knowledge-Based Probabilistic Collaborative Representation for Visual Recognition , 2020, IEEE Transactions on Cybernetics.

[35]  Mohit Bansal,et al.  Polite Dialogue Generation Without Parallel Data , 2018, TACL.

[36]  Xiaobo Wu,et al.  A Method of Emotional Analysis of Movie Based on Convolution Neural Network and Bi-directional LSTM RNN , 2017, 2017 IEEE Second International Conference on Data Science in Cyberspace (DSC).

[37]  Yue Zhang,et al.  Context-Sensitive Lexicon Features for Neural Sentiment Analysis , 2016, EMNLP.

[38]  Hongyu Guo,et al.  Generating Text with Deep Reinforcement Learning , 2015, ArXiv.

[39]  Matthew R. Walter,et al.  Coherent Dialogue with Attention-Based Language Models , 2016, AAAI.

[40]  Xiaoyan Zhu,et al.  Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory , 2017, AAAI.

[41]  Zhenrong Deng,et al.  A Two-stage Chinese text summarization algorithm using keyword information and adversarial learning , 2020, Neurocomputing.

[42]  Ting Liu,et al.  Generating Reasonable and Diversified Story Ending Using Sequence to Sequence Model with Adversarial Training , 2018, COLING.

[43]  David Suendermann-Oeft,et al.  From rule-based to statistical grammars: Continuous improvement of large-scale spoken dialog systems , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[44]  Marilyn A. Walker,et al.  Trainable Sentence Planning for Complex Information Presentations in Spoken Dialog Systems , 2004, ACL.

[45]  Hao Tian,et al.  Policy Learning for Domain Selection in an Extensible Multi-domain Spoken Dialogue System , 2014, EMNLP.

[46]  Matthew Henderson,et al.  Deep Neural Network Approach for the Dialog State Tracking Challenge , 2013, SIGDIAL Conference.

[47]  Erik Cambria,et al.  A Review of Sentiment Analysis Research in Chinese Language , 2017, Cognitive Computation.

[48]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[49]  Antoine Raux,et al.  The Dialog State Tracking Challenge , 2013, SIGDIAL Conference.

[50]  Quoc V. Le,et al.  A Neural Conversational Model , 2015, ArXiv.