SARG: A Novel Semi Autoregressive Generator for Multi-turn Incomplete Utterance Restoration

Dialogue systems in the open domain have achieved great success due to large conversation data and the development of deep learning, but multi-turn scenarios are still a challenge because of the frequent coreference and information omission. In this paper, we investigate the incomplete utterance restoration since it has brought general improvement over multi-turn dialogue systems in recent studies. Inspired by the autoregression for generation and the sequence labeling for text editing, we propose a novel semi autoregressive generator (SARG) with the high efficiency and flexibility. Moreover, experiments on Restoration-200k show that our proposed model significantly outperforms the state-of-the-art models with faster inference speed.

[1]  Wei Wu,et al.  Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks , 2020, EMNLP.

[2]  Raymond Hendy Susanto,et al.  The CoNLL-2014 Shared Task on Grammatical Error Correction , 2014 .

[3]  Rui Yan,et al.  Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System , 2016, SIGIR.

[4]  Bowen Wu,et al.  Ranking Responses Oriented to Conversational Relevance in Chat-bots , 2016, COLING.

[5]  Cheng Niu,et al.  Improving Multi-turn Dialogue Modelling with Utterance ReWriter , 2019, ACL.

[6]  Quoc V. Le,et al.  Towards a Human-like Open-Domain Chatbot , 2020, ArXiv.

[7]  Zhoujun Li,et al.  Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots , 2016, ArXiv.

[8]  Wei Zhao,et al.  Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data , 2019, NAACL.

[9]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[10]  Jianfeng Gao,et al.  Challenges in Building Intelligent Open-domain Dialog Systems , 2019, ACM Trans. Inf. Syst..

[11]  Ming Zhou,et al.  Fluency Boost Learning and Inference for Neural Grammatical Error Correction , 2018, ACL.

[12]  Ian Beaver,et al.  A Case Study of User Communication Styles with Customer Service Agents versus Intelligent Virtual Agents , 2020, SIGdial.

[13]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[14]  Hai Zhao,et al.  Modeling Multi-turn Conversation with Deep Utterance Aggregation , 2018, COLING.

[15]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[16]  Kathleen McKeown,et al.  Supervised Sentence Fusion with Single-Stage Inference , 2013, IJCNLP.

[17]  Hang Li,et al.  An Information Retrieval Approach to Short Text Conversation , 2014, ArXiv.

[18]  Emiel Krahmer,et al.  Sentence Simplification by Monolingual Machine Translation , 2012, ACL.

[19]  Shamil Chollampatt,et al.  Neural Quality Estimation of Grammatical Error Correction , 2018, EMNLP.

[20]  Xia Gong,et al.  Customer Service Automatic Answering System Based on Natural Language Processing , 2019, SSPS 2019.

[21]  Xiaodong Liu,et al.  Unified Language Model Pre-training for Natural Language Understanding and Generation , 2019, NeurIPS.

[22]  Bowen Zhou,et al.  Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation , 2016, AAAI.

[23]  Wanxiang Che,et al.  Pre-Training with Whole Word Masking for Chinese BERT , 2019, ArXiv.

[24]  Piji Li,et al.  An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation , 2020, ArXiv.

[25]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[26]  Furu Wei,et al.  Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization , 2018, ACL.

[27]  Franck Dernoncourt,et al.  Analyzing Sentence Fusion in Abstractive Summarization , 2019, EMNLP.

[28]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[29]  Yen-Chun Chen,et al.  Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting , 2018, ACL.

[30]  Axel-Cyrille Ngonga Ngomo,et al.  Enhancing Community Interactions with Data-Driven Chatbots--The DBpedia Chatbot , 2018, WWW.

[31]  Wei-Ying Ma,et al.  Topic Aware Neural Response Generation , 2016, AAAI.

[32]  Aliaksei Severyn,et al.  Encode, Tag, Realize: High-Precision Text Editing , 2019, EMNLP.

[33]  Xuan Liu,et al.  Multi-view Response Selection for Human-Computer Conversation , 2016, EMNLP.

[34]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[35]  Ying Chen,et al.  Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network , 2018, ACL.

[36]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[37]  Wei-Ying Ma,et al.  Topic Augmented Neural Response Generation with a Joint Attention Mechanism , 2016, ArXiv.

[38]  Alan Ritter,et al.  Data-Driven Response Generation in Social Media , 2011, EMNLP.

[39]  Yan Wang,et al.  Improving Open-Domain Dialogue Systems via Multi-Turn Incomplete Utterance Restoration , 2019, EMNLP.

[40]  Fei Lin,et al.  A Hierarchical Structured Multi-Head Attention Network for Multi-Turn Response Generation , 2020, IEEE Access.

[41]  Mirella Lapata,et al.  Sentence Simplification with Deep Reinforcement Learning , 2017, EMNLP.