Reference Language based Unsupervised Neural Machine Translation

Exploiting common language as an auxiliary for better translation has a long tradition in machine translation, which lets supervised learning based machine translation enjoy the enhancement delivered by the well-used pivot language, in case that the prerequisite of parallel corpus from source language to target language cannot be fully satisfied. The rising of unsupervised neural machine translation (UNMT) seems completely relieving the parallel corpus curse, though still subject to unsatisfactory performance so far due to vague clues available used for its core back-translation training. Further enriching the idea of pivot translation by freeing the use of parallel corpus other than its specified source and target, we propose a new reference language based UNMT framework, in which the reference language only shares parallel corpus with the source, indicating clear enough signal to help the reconstruction training of UNMT through a proposed reference agreement mechanism. Experimental results show that our methods improve the quality of UNMT over that of a strong baseline in terms of only one auxiliary language, demonstrating the usefulness of the proposed reference language based UNMT with a good start.

[1]  Philipp Koehn,et al.  Clause Restructuring for Statistical Machine Translation , 2005, ACL.

[2]  Hitoshi Isahara,et al.  A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation , 2007, NAACL.

[3]  Hai Zhao,et al.  Explicit Sentence Compression for Neural Machine Translation , 2019, AAAI.

[4]  Hua Wu,et al.  Pivot language approach for phrase-based statistical machine translation , 2007, ACL.

[5]  Hai Zhao,et al.  Semantics-aware BERT for Language Understanding , 2020, AAAI.

[6]  Hermann Ney,et al.  Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages , 2019, EMNLP.

[7]  Marjan Ghazvininejad,et al.  Multilingual Denoising Pre-training for Neural Machine Translation , 2020, Transactions of the Association for Computational Linguistics.

[8]  Ankur Bapna,et al.  The Missing Ingredient in Zero-Shot Neural Machine Translation , 2019, ArXiv.

[9]  Thibault Sellam,et al.  A Multilingual View of Unsupervised Machine Translation , 2020, FINDINGS.

[10]  Eneko Agirre,et al.  Unsupervised Neural Machine Translation , 2017, ICLR.

[11]  Mark Steedman,et al.  A massively parallel corpus: the Bible in 100 languages , 2014, Lang. Resour. Evaluation.

[12]  Hai Zhao,et al.  Syntax for Semantic Role Labeling, To Be, Or Not To Be , 2018, ACL.

[13]  Tie-Yan Liu,et al.  Dual Learning for Machine Translation , 2016, NIPS.

[14]  Yann Dauphin,et al.  Convolutional Sequence to Sequence Learning , 2017, ICML.

[15]  Tiejun Zhao,et al.  Unsupervised Bilingual Word Embedding Agreement for Unsupervised Neural Machine Translation , 2019, ACL.

[16]  Ahmed Abdelali,et al.  The AMARA Corpus: Building Parallel Language Resources for the Educational Domain , 2014, LREC.

[17]  Hai Zhao,et al.  A Unified Syntax-aware Framework for Semantic Role Labeling , 2018, EMNLP.

[18]  Xu Tan,et al.  Unsupervised Pivot Translation for Distant Languages , 2019, ACL.

[19]  Nenghai Yu,et al.  Dual Supervised Learning , 2017, ICML.

[20]  Xu Tan,et al.  MASS: Masked Sequence to Sequence Pre-training for Language Generation , 2019, ICML.

[21]  Flemming Topsøe,et al.  Jensen-Shannon divergence and Hilbert space embedding , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[22]  Jörg Tiedemann,et al.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles , 2016, LREC.

[23]  Min Zhang,et al.  Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation , 2019, AAAI.

[24]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[25]  Guillaume Lample,et al.  Phrase-Based & Neural Unsupervised Machine Translation , 2018, EMNLP.

[26]  Philipp Koehn,et al.  Six Challenges for Neural Machine Translation , 2017, NMT@ACL.

[27]  Tiejun Zhao,et al.  Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation , 2020, ACL.

[28]  Rico Sennrich,et al.  Edinburgh Neural Machine Translation Systems for WMT 16 , 2016, WMT.

[29]  Hai Zhao,et al.  Data-dependent Gaussian Prior Objective for Language Generation , 2020, ICLR.

[30]  Hai Zhao,et al.  Unsupervised Neural Machine Translation with Indirect Supervision , 2020, ArXiv.

[31]  Jörg Tiedemann,et al.  Parallel Data, Tools and Interfaces in OPUS , 2012, LREC.

[32]  Eneko Agirre,et al.  Learning bilingual word embeddings with (almost) no bilingual data , 2017, ACL.

[33]  Hai Zhao,et al.  A Full End-to-End Semantic Role Labeler, Syntactic-agnostic Over Syntactic-aware? , 2018, COLING.

[34]  Orhan Firat,et al.  Massively Multilingual Neural Machine Translation , 2019, NAACL.

[35]  Nenghai Yu,et al.  Dual Inference for Machine Learning , 2017, IJCAI.

[36]  Satoshi Nakamura,et al.  On the Importance of Pivot Language Selection for Statistical Machine Translation , 2009, NAACL.

[37]  Ankur P. Parikh,et al.  Consistency by Agreement in Zero-Shot Neural Machine Translation , 2019, NAACL.

[38]  Zhuosheng Zhang,et al.  Effective Subword Segmentation for Text Comprehension , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[39]  Guillaume Lample,et al.  Word Translation Without Parallel Data , 2017, ICLR.

[40]  Andy Way,et al.  Pivot Machine Translation Using Chinese as Pivot Language , 2018, Communications in Computer and Information Science.

[41]  Junru Zhou,et al.  Parsing All: Syntax and Semantics, Dependencies and Spans , 2020, EMNLP.

[42]  Junru Zhou,et al.  Head-Driven Phrase Structure Grammar Parsing on Penn Treebank , 2019, ACL.

[43]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[44]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[45]  Guillaume Lample,et al.  Unsupervised Machine Translation Using Monolingual Corpora Only , 2017, ICLR.

[46]  Marcin Junczys-Dowmunt,et al.  The United Nations Parallel Corpus v1.0 , 2016, LREC.

[47]  Guillaume Lample,et al.  Cross-lingual Language Model Pretraining , 2019, NeurIPS.

[48]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[49]  Yang Liu,et al.  Joint Training for Pivot-based Neural Machine Translation , 2016, IJCAI.