Symmetrical Adversarial Training Network: A Novel Model for Text Generation

Text generation has always been the core issue in the field of natural language processing. Over the past decades, Generative Adversarial network (GAN) has proven its great potential in generating realistic synthetic data, performing competitively in various domains like computer vision. However, the characteristics of text discretization limit the application of GANs in natural language processing. In this paper, we proposed a novel Symmetrical Adversarial Training Network (SATN) which employed symmetrical text comparison mechanism for the purpose of generating more realistic and coherent text samples. In the SATN, a Deep Attention Similarity Model (DASM) was designed to extract fine-grained original-synthetic sentence feature match loss for improving the performance of generative network. With DASM, the SATN can identify the difference between sentences in word level and pay attention to relevant meaningful words. Meanwhile, we utilize the DASM loss to compensate for the defect of the objective function in adversarial training. Our experiments demonstrated significant improvement in evaluation.

[1]  Lantao Yu,et al.  SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient , 2016, AAAI.

[2]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jason Baldridge,et al.  Supervised Text-based Geolocation Using Language Models on an Adaptive Grid , 2012, EMNLP.

[4]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[5]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[6]  Alan Ritter,et al.  Adversarial Learning for Neural Dialogue Generation , 2017, EMNLP.

[7]  Ruslan Salakhutdinov,et al.  Learning Deep Generative Models , 2009 .

[8]  S. Johansen,et al.  MAXIMUM LIKELIHOOD ESTIMATION AND INFERENCE ON COINTEGRATION — WITH APPLICATIONS TO THE DEMAND FOR MONEY , 2009 .

[9]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[10]  Matt J. Kusner,et al.  GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution , 2016, ArXiv.

[11]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[12]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[13]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[14]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[15]  Zhi Chen,et al.  Adversarial Feature Matching for Text Generation , 2017, ICML.

[16]  Kathleen R. McKeown,et al.  Text generation , 1985 .

[17]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[18]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[19]  Kevin Lin,et al.  Adversarial Ranking for Language Generation , 2017, NIPS.

[20]  Benoit Lavoie,et al.  A Fast and Portable Realizer for Text Generation Systems , 1997, ANLP.

[21]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[22]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[23]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[24]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[25]  Martin A. Riedmiller,et al.  Deep auto-encoder neural networks in reinforcement learning , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).