Revisiting Multi-Domain Machine Translation

When building machine translation systems, one often needs to make the best out of heterogeneous sets of parallel data in training, and to robustly handle inputs from unexpected domains in testing. This multi-domain scenario has attracted a lot of recent work that fall under the general umbrella of transfer learning. In this study, we revisit multi-domain machine translation, with the aim to formulate the motivations for developing such systems and the associated expectations with respect to performance. Our experiments with a large sample of multi-domain systems show that most of these expectations are hardly met and suggest that further work is needed to better analyze the current behaviour of multi-domain systems and to make them fully hold their promises.

[1]  Yishay Mansour,et al.  Domain Adaptation with Multiple Sources , 2008, NIPS.

[2]  Josep Maria Crego,et al.  Domain Control for Neural Machine Translation , 2016, RANLP.

[3]  Marcello Federico,et al.  Multi-Domain Neural Machine Translation through Unsupervised Adaptation , 2017, WMT.

[4]  Ankur Bapna,et al.  Simple, Scalable Adaptation for Neural Machine Translation , 2019, EMNLP.

[5]  Philipp Koehn,et al.  Statistical Significance Tests for Machine Translation Evaluation , 2004, EMNLP.

[6]  Jiajun Zhang,et al.  One Sentence One Model for Neural Machine Translation , 2018, LREC.

[7]  Philipp Koehn,et al.  Dynamic Topic Adaptation for Phrase-based MT , 2014, EACL.

[8]  Percy Liang,et al.  Distributionally Robust Language Modeling , 2019, EMNLP.

[9]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[10]  Marcello Federico,et al.  Domain Adaptation for Statistical Machine Translation with Monolingual Resources , 2009, WMT@EACL.

[11]  Chenhui Chu,et al.  Multilingual and Multi-Domain Adaptation for Neural Machine Translation , 2018 .

[12]  Jianfeng Gao,et al.  Domain Adaptation via Pseudo In-Domain Data Selection , 2011, EMNLP.

[13]  Graham Neubig,et al.  Extreme Adaptation for Personalized Neural Machine Translation , 2018, ACL.

[14]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[15]  Markus Freitag,et al.  Fast Domain Adaptation for Neural Machine Translation , 2016, ArXiv.

[16]  ChengXiang Zhai,et al.  Instance Weighting for Domain Adaptation in NLP , 2007, ACL.

[17]  Jörg Tiedemann,et al.  News from OPUS — A collection of multilingual parallel corpora with tools and interfaces , 2009 .

[18]  Bill Byrne,et al.  UCAM Biomedical Translation at WMT19: Transfer Learning Multi-domain Ensembles , 2019, WMT.

[19]  Tuo Zhao,et al.  Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing , 2019, ACL.

[20]  Daniel Marcu,et al.  Domain Adaptation for Statistical Classifiers , 2006, J. Artif. Intell. Res..

[21]  Quoc V. Le,et al.  Effective Domain Mixing for Neural Machine Translation , 2017, WMT.

[22]  Mauro Cettolo,et al.  WIT3: Web Inventory of Transcribed and Translated Talks , 2012, EAMT.

[23]  Koby Crammer,et al.  A theory of learning from different domains , 2010, Machine Learning.

[24]  Marine Carpuat,et al.  Multi-Task Neural Models for Translating Between Styles Within and Across Languages , 2018, COLING.

[25]  Jörg Tiedemann,et al.  Parallel Data, Tools and Interfaces in OPUS , 2012, LREC.

[26]  Yang Liu,et al.  Exploring Discriminative Word-Level Domain Contexts for Multi-Domain Neural Machine Translation , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Alexander M. Rush,et al.  OpenNMT: Open-Source Toolkit for Neural Machine Translation , 2017, ACL.

[28]  Andy Way,et al.  Topic-Informed Neural Machine Translation , 2016, COLING.

[29]  Marcello Federico,et al.  Neural vs. Phrase-Based Machine Translation in a Multi-Domain Scenario , 2017, EACL.

[30]  Mark Fishel,et al.  Multi-Domain Neural Machine Translation , 2018, EAMT.

[31]  François Yvon,et al.  Generic and Specialized Word Embeddings for Multi-Domain Machine Translation , 2019, IWSLT.

[32]  Rui Wang,et al.  A Survey of Domain Adaptation for Neural Machine Translation , 2018, COLING.

[33]  Yoshua Bengio,et al.  Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism , 2016, NAACL.

[34]  John Blitzer,et al.  Domain adaptation of natural language processing systems , 2008 .

[35]  Rico Sennrich,et al.  A Multi-Domain Translation Model Framework for Statistical Machine Translation , 2013, ACL.

[36]  Graham Neubig,et al.  compare-mt: A Tool for Holistic Comparison of Language Generation Systems , 2019, NAACL.

[37]  Yang Liu,et al.  Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination , 2018, EMNLP.

[38]  Roland Kuhn,et al.  Mixture-Model Adaptation for SMT , 2007, WMT@ACL.

[39]  Rico Sennrich,et al.  Controlling Politeness in Neural Machine Translation via Side Constraints , 2016, NAACL.

[40]  Vladimir Eidelman,et al.  Topic Models for Dynamic Translation Model Adaptation , 2012, ACL.

[41]  Alexandra Birch,et al.  Mixed domain vs. multi-domain statistical machine translation , 2015, MTSUMMIT.

[42]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[43]  Carolyn Penstein Rosé,et al.  Multi-Domain Learning: When Do Domains Matter? , 2012, EMNLP-CoNLL.

[44]  Christopher D. Manning,et al.  Hierarchical Bayesian Domain Adaptation , 2009, NAACL.

[45]  Koby Crammer,et al.  Online Methods for Multi-Domain Learning and Adaptation , 2008, EMNLP.

[46]  Andy Way,et al.  Combining Multi-Domain Statistical Machine Translation Models using Automatic Classifiers , 2010, AMTA.

[47]  Yishay Mansour,et al.  Multiple Source Adaptation and the Rényi Divergence , 2009, UAI.

[48]  Martin Wattenberg,et al.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation , 2016, TACL.

[49]  Tomaz Erjavec,et al.  The JRC-Acquis: A Multilingual Aligned Parallel Corpus with 20+ Languages , 2006, LREC.

[50]  Wenhu Chen,et al.  Guided Alignment Training for Topic-Aware Neural Machine Translation , 2016, AMTA.

[51]  Jean Senellart,et al.  Lexical Micro-adaptation for Neural Machine Translation , 2019, IWSLT.

[52]  Dragos Stefan Munteanu,et al.  Measuring Machine Translation Errors in New Domains , 2013, TACL.

[53]  Amr Sharaf,et al.  Meta-Learning for Few-Shot NMT Adaptation , 2020, NGT.

[54]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[55]  Mehryar Mohri,et al.  Algorithms and Theory for Multiple-Source Adaptation , 2018, NeurIPS.

[56]  Rico Sennrich,et al.  Regularization techniques for fine-tuning in neural machine translation , 2017, EMNLP.

[57]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .

[58]  Christopher D. Manning,et al.  Stanford Neural Machine Translation Systems for Spoken Language Domains , 2015, IWSLT.

[59]  Alon Lavie,et al.  One System, Many Domains: Open-Domain Statistical Machine Translation via Feature Augmentation , 2012, AMTA.