A Submodular Optimization-Based VAE-Transformer Framework for Paraphrase Generation

Paraphrase plays an important role in various Natural Language Processing (NLP) problems, such as question answering, information retrieval, conversation systems, etc. Previous approaches mainly concentrate on producing paraphrases with similar semantics, namely fidelity, while recent ones begin to focus on the diversity of generated paraphrases. However, most of the existing models fail to explicitly emphasize on both metrics above. To fill this gap, we propose a submodular optimization-based VAE-transformer model to generate more consistent and diverse phrases. Through extensive experiments on datasets like Quora and Twitter, we demonstrate that our proposed model outperforms state-of-the-art baselines on BLEU, METEOR, TERp and n-distinct grams. Furthermore, through ablation study, our results suggest that incorporating VAE and submodularity functions could effectively promote fidelity and diversity respectively.

[1]  Furu Wei,et al.  Dictionary-Guided Editing Networks for Paraphrase Generation , 2018, AAAI.

[2]  Dongyan Zhao,et al.  Towards a Neural Conversation Model With Diversity Net Using Determinantal Point Processes , 2018, AAAI.

[3]  Ankush Gupta,et al.  A Deep Generative Framework for Paraphrase Generation , 2017, AAAI.

[4]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[5]  Luke S. Zettlemoyer,et al.  Adversarial Example Generation with Syntactically Controlled Paraphrase Networks , 2018, NAACL.

[6]  Gregory Shakhnarovich,et al.  A Systematic Exploration of Diversity in Machine Translation , 2013, EMNLP.

[7]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[8]  Danyang Liu,et al.  A Transformer-Based Variational Autoencoder for Sentence Generation , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[9]  Ben Taskar,et al.  Determinantal Point Processes for Machine Learning , 2012, Found. Trends Mach. Learn..

[10]  Alexander F. Gelbukh,et al.  Synonymous Paraphrasing Using WordNet and Internet , 2004, NLDB.

[11]  Chris Quirk,et al.  Monolingual Machine Translation for Paraphrase Generation , 2004, EMNLP.

[12]  Hang Li,et al.  Paraphrase Generation with Deep Reinforcement Learning , 2017, EMNLP.

[13]  Oladimeji Farri,et al.  Neural Paraphrase Generation with Stacked Residual LSTM Networks , 2016, COLING.

[14]  Regina Barzilay,et al.  Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment , 2003, NAACL.

[15]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..

[16]  Andreas Krause,et al.  Efficient Minimization of Decomposable Submodular Functions , 2010, NIPS.

[17]  Mirella Lapata,et al.  Sentence Simplification with Deep Reinforcement Learning , 2017, EMNLP.

[18]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[19]  Kathleen McKeown,et al.  Paraphrasing Questions Using Given and new information , 1983, CL.

[20]  S. Shankar Sastry,et al.  Dissimilarity-Based Sparse Subset Selection , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Jianfeng Gao,et al.  A Diversity-Promoting Objective Function for Neural Conversation Models , 2015, NAACL.

[22]  Ashwin K. Vijayakumar,et al.  Diverse Beam Search for Improved Description of Complex Scenes , 2018, AAAI.

[23]  Hua He,et al.  A Continuously Growing Dataset of Sentential Paraphrases , 2017, EMNLP.

[24]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[25]  Partha Talukdar,et al.  Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation , 2019, NAACL.

[26]  Jakob Uszkoreit,et al.  A Decomposable Attention Model for Natural Language Inference , 2016, EMNLP.

[27]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.