IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation

Text attribute transfer aims to automatically rewrite sentences such that they possess certain linguistic attributes, while simultaneously preserving their semantic content. This task remains challenging due to a lack of supervised parallel data. Existing approaches try to explicitly disentangle content and attribute information, but this is difficult and often results in poor content-preservation and ungrammaticality. In contrast, we propose a simpler approach, Iterative Matching and Translation (IMaT), which: (1) constructs a pseudo-parallel corpus by aligning a subset of semantically similar sentences from the source and the target corpora; (2) applies a standard sequence-to-sequence model to learn the attribute transfer; (3) iteratively improves the learned transfer function by refining imperfections in the alignment. In sentiment modification and formality transfer tasks, our method outperforms complex state-of-the-art systems by a large margin. As an auxiliary contribution, we produce a publicly-available test set with human-generated transfer references.

[1]  Alexander M. Rush,et al.  OpenNMT: Open-Source Toolkit for Neural Machine Translation , 2017, ACL.

[2]  Guillaume Lample,et al.  Multiple-Attribute Text Style Transfer , 2018, ArXiv.

[3]  Mohit Bansal,et al.  Polite Dialogue Generation Without Parallel Data , 2018, TACL.

[4]  Eric P. Xing,et al.  Toward Controlled Generation of Text , 2017, ICML.

[5]  Guillaume Lample,et al.  Unsupervised Machine Translation Using Monolingual Corpora Only , 2017, ICLR.

[6]  Matt J. Kusner,et al.  Supervised Word Mover's Distance , 2016, NIPS.

[7]  Nan Hua,et al.  Universal Sentence Encoder , 2018, ArXiv.

[8]  Diana Inkpen,et al.  Generation of Formal and Informal Sentences , 2011, ENLG.

[9]  Joel R. Tetreault,et al.  An Empirical Analysis of Formality in Online Communication , 2016, TACL.

[10]  Srinivas Bangalore,et al.  Trainable Speaker-Based Referring Expression Generation , 2008, CoNLL.

[11]  Samy Bengio,et al.  Content preserving text generation with attribute controls , 2018, NeurIPS.

[12]  Philipp Koehn,et al.  Statistical Significance Tests for Machine Translation Evaluation , 2004, EMNLP.

[13]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[14]  Guillaume Lample,et al.  Phrase-Based & Neural Unsupervised Machine Translation , 2018, EMNLP.

[15]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[16]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[17]  Kristina Toutanova,et al.  Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment , 2010, NAACL.

[18]  Marilyn A. Walker,et al.  Controlling Personality-Based Stylistic Variation with Neural Natural Language Generators , 2018, SIGDIAL Conference.

[19]  Leo Wanner,et al.  Data-driven sentence generation with non-isomorphic trees , 2015, HLT-NAACL.

[20]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[21]  Harsh Jhamtani,et al.  Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models , 2017, Proceedings of the Workshop on Stylistic Variation.

[22]  Enhong Chen,et al.  Style Transfer as Unsupervised Machine Translation , 2018, ArXiv.

[23]  Ion Androutsopoulos,et al.  Using Centroids of Word Embeddings and Word Mover’s Distance for Biomedical Document Retrieval in Question Answering , 2016, BioNLP@ACL.

[24]  Matt J. Kusner,et al.  From Word Embeddings To Document Distances , 2015, ICML.

[25]  Joel R. Tetreault,et al.  Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer , 2018, NAACL.

[26]  Cícero Nogueira dos Santos,et al.  Improved Neural Text Attribute Transfer with Non-parallel Data , 2017, ArXiv.

[27]  Alexander M. Fraser,et al.  Improved Machine Translation Performance via Parallel Sentence Extraction from Comparable Corpora , 2004, NAACL.

[28]  Michael Gamon,et al.  Obfuscating Document Stylometry to Preserve Author Anonymity , 2006, ACL.

[29]  Percy Liang,et al.  Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer , 2018, NAACL.

[30]  Zhou Yu,et al.  Structured Content Preservation for Unsupervised Text Style Transfer , 2018, ArXiv.

[31]  Tommi S. Jaakkola,et al.  Sequence to Better Sequence: Continuous Revision of Combinatorial Structures , 2017, ICML.

[32]  Guillaume Lample,et al.  Word Translation Without Parallel Data , 2017, ICLR.

[33]  Yulia Tsvetkov,et al.  Style Transfer Through Back-Translation , 2018, ACL.

[34]  Dumitru Erhan,et al.  Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Nasser M. Nasrabadi,et al.  Unsupervised Image-to-Image Translation Using Domain-Specific Variational Information Bound , 2018, NeurIPS.

[36]  Eneko Agirre,et al.  Unsupervised Neural Machine Translation , 2017, ICLR.

[37]  Cícero Nogueira dos Santos,et al.  Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer , 2018, ACL.

[38]  Pascale Fung,et al.  An IR Approach for Translating New Words from Nonparallel, Comparable Texts , 1998, ACL.

[39]  Regina Barzilay,et al.  Style Transfer from Non-Parallel Text by Cross-Alignment , 2017, NIPS.

[40]  Patrick Juola,et al.  Analyzing Stylometric Approaches to Author Obfuscation , 2011, IFIP Int. Conf. Digital Forensics.

[41]  Kwang In Kim,et al.  Unsupervised Attention-guided Image to Image Translation , 2018, NeurIPS.

[42]  Jackie Chi Kit Cheung,et al.  Stylistic Transfer in Natural Language Generation Systems Using Recurrent Neural Networks , 2016 .

[43]  Tong Wang,et al.  Automatic Acquisition of Lexical Formality , 2010, COLING.

[44]  Emiel Krahmer,et al.  Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation , 2017, J. Artif. Intell. Res..

[45]  Guillaume Lample,et al.  Multiple-Attribute Text Rewriting , 2018, ICLR.

[46]  Emiel Krahmer,et al.  Computational Generation of Referring Expressions: A Survey , 2012, CL.

[47]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[48]  Sanja Fidler,et al.  Skip-Thought Vectors , 2015, NIPS.

[49]  Dongyan Zhao,et al.  Style Transfer in Text: Exploration and Evaluation , 2017, AAAI.

[50]  Albert Gatt,et al.  From data to text in the Neonatal Intensive Care Unit: Using NLG technology for decision support and information management , 2009, AI Commun..

[51]  Marilyn A. Walker,et al.  Controlling User Perceptions of Linguistic Style: Trainable Generation of Personality Traits , 2011, CL.

[52]  Holger Schwenk,et al.  Supervised Learning of Universal Sentence Representations from Natural Language Inference Data , 2017, EMNLP.

[53]  Christian S. Perone,et al.  Evaluation of sentence embeddings in downstream and linguistic probing tasks , 2018, ArXiv.

[54]  Cordelia Schmid,et al.  Unsupervised Learning of Artistic Styles with Archetypal Style Analysis , 2018, NeurIPS.

[55]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.