Gender bias amplification during Speed-Quality optimization in Neural Machine Translation

Is bias amplified when neural machine translation (NMT) models are optimized for speed and evaluated on generic test sets using BLEU? We investigate architectures and techniques commonly used to speed up decoding in Transformer-based models, such as greedy search, quantization, average attention networks (AANs) and shallow decoder models and show their effect on gendered noun translation. We construct a new gender bias test set, SimpleGEN, based on gendered noun phrases in which there is a single, unambiguous, correct answer. While we find minimal overall BLEU degradation as we apply speed optimizations, we observe that gendered noun translation performance degrades at a much faster rate.

[1]  Matteo Negri,et al.  Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus , 2020, ACL.

[2]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[3]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[4]  Noah A. Smith,et al.  Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation , 2020, ArXiv.

[5]  Deyi Xiong,et al.  Accelerating Neural Transformer via an Average Attention Network , 2018, ACL.

[6]  Miguel Ballesteros,et al.  Pieces of Eight: 8-bit Neural Machine Translation , 2018, NAACL.

[7]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[8]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[9]  Preslav Nakov,et al.  Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction , 2016, WMT.

[10]  Marta R. Costa-jussà,et al.  Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques , 2019, Proceedings of the First Workshop on Gender Bias in Natural Language Processing.

[11]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[12]  Andy Way,et al.  Getting Gender Right in Neural Machine Translation , 2019, EMNLP.

[13]  M. Costa-jussà,et al.  Fine-tuning Neural Machine Translation on Gender-Balanced Datasets , 2020, GEBNLP.

[14]  Bill Byrne,et al.  Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem , 2020, ACL.

[15]  Noah A. Smith,et al.  Evaluating Gender Bias in Machine Translation , 2019, ACL.

[16]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[17]  Rico Sennrich,et al.  Deep architectures for Neural Machine Translation , 2017, WMT.

[18]  B. Byrne,et al.  Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It , 2020, GEBNLP.

[19]  Ryan Cotterell,et al.  Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology , 2019, ACL.

[20]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[21]  Marcis Pinnis,et al.  Mitigating Gender Bias in Machine Translation with Target Gender Annotations , 2020, WMT.

[22]  Carlos Escolano,et al.  Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters , 2020, ArXiv.

[23]  Zeyu Li,et al.  Learning Gender-Neutral Word Embeddings , 2018, EMNLP.

[24]  Jingbo Zhu,et al.  Learning Deep Transformer Models for Machine Translation , 2019, ACL.

[25]  Moshe Wasserblat,et al.  Q8BERT: Quantized 8Bit BERT , 2019, 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS).

[26]  Ioannis Konstas,et al.  Findings of the Fourth Workshop on Neural Generation and Translation , 2020, NGT@ACL.

[27]  Marcin Junczys-Dowmunt,et al.  From Research to Production and Back: Ludicrously Fast Neural Machine Translation , 2019, EMNLP.