Improving Low-Resource NMT through Relevance Based Linguistic Features Incorporation

In this study, linguistic knowledge at different levels are incorporated into the neural machine translation (NMT) framework to improve translation quality for language pairs with extremely limited data. Integrating manually designed or automatically extracted features into the NMT framework is known to be beneficial. However, this study emphasizes that the relevance of the features is crucial to the performance. Specifically, we propose two methods, 1) self relevance and 2) word-based relevance, to improve the representation of features for NMT. Experiments are conducted on translation tasks from English to eight Asian languages, with no more than twenty thousand sentences for training. The proposed methods improve translation quality for all tasks by up to 3.09 BLEU points. Discussions with visualization provide the explainability of the proposed methods where we show that the relevance methods provide weights to features thereby enhancing their impact on low-resource machine translation.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Naoaki Okazaki,et al.  Enhancing Machine Translation with Dependency-Aware Self-Attention , 2020, ACL.

[3]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[4]  Nan Yang,et al.  Dependency-to-Dependency Neural Machine Translation , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[5]  Lukasz Kaiser,et al.  Reformer: The Efficient Transformer , 2020, ICLR.

[6]  François Yvon,et al.  Word Representations in Factored Neural Machine Translation , 2017, WMT.

[7]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[8]  Fethi Bougares,et al.  Factored Neural Machine Translation , 2016, ArXiv.

[9]  Guodong Zhou,et al.  Modeling Source Syntax for Neural Machine Translation , 2017, ACL.

[10]  Gholamreza Haffari,et al.  Improving Neural Translation Models with Linguistic Factors , 2016, ALTA.

[11]  Meishan Zhang,et al.  Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations , 2019, NAACL.

[12]  Masao Utiyama,et al.  NOVA , 2018, ACM Trans. Asian Low Resour. Lang. Inf. Process..

[13]  Shujian Huang,et al.  Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder , 2017, ACL.

[14]  Jingbo Zhu,et al.  Linguistic Knowledge-Aware Neural Machine Translation , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[15]  Yoshimasa Tsuruoka,et al.  Neural Machine Translation with Source-Side Latent Graph Parsing , 2017, EMNLP.

[16]  Yoshimasa Tsuruoka,et al.  Tree-to-Sequence Attentional Neural Machine Translation , 2016, ACL.

[17]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[18]  Rico Sennrich,et al.  Linguistic Input Features Improve Neural Machine Translation , 2016, WMT.

[19]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[20]  Yating Yang,et al.  Dual-Source Transformer Model for Neural Machine Translation with Linguistic Knowledge , 2020 .

[21]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[22]  Khalil Sima'an,et al.  Graph Convolutional Encoders for Syntax-aware Neural Machine Translation , 2017, EMNLP.

[23]  Fethi Bougares,et al.  Factored Neural Machine Translation Architectures , 2016, IWSLT.

[24]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[25]  Xing Shi,et al.  Does String-Based Neural MT Learn Source Syntax? , 2016, EMNLP.

[26]  Masao Utiyama,et al.  Introduction of the Asian Language Treebank , 2016, 2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques (O-COCOSDA).