Repairing Pronouns in Translation with BERT-Based Post-Editing

Pronouns are important determinants of a text’s meaning but difficult to translate. This is because pronoun choice can depend on entities described in previous sentences, and in some languages pronouns may be dropped when the referent is inferrable from the context. These issues can lead Neural Machine Translation (NMT) systems to make critical errors on pronouns that impair intelligibility and even reinforce gender bias. We investigate the severity of this pronoun issue, showing that (1) in some domains, pronoun choice can account for more than half of a NMT systems’ errors, and (2) pronouns have a disproportionately large impact on perceived translation quality. We then investigate a possible solution: finetuning BERT on a pronoun prediction task using chunks of source-side sentences, then using the resulting classifier to repair the translations of an existing NMT model. We offer an initial case study of this approach for the Japanese-English language pair, observing that a small number of translations are significantly improved according to human evaluators.

[1]  Preslav Nakov,et al.  Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction , 2016, WMT.

[2]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[3]  Andy Way,et al.  Getting Gender Right in Neural Machine Translation , 2019, EMNLP.

[4]  Hugo Mailhot,et al.  MorphoLex: A derivational morphological database for 70,000 English words , 2018, Behavior research methods.

[5]  Jörg Tiedemann,et al.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles , 2016, LREC.

[6]  Alexander M. Fraser,et al.  Improving Anaphora Resolution in Neural Machine Translation Using Curriculum Learning , 2019, MTSummit.

[7]  Xing Wang,et al.  Exploiting Sentential Context for Neural Machine Translation , 2019, ACL.

[8]  Quoc V. Le,et al.  Effective Domain Mixing for Neural Machine Translation , 2017, WMT.

[9]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[10]  Andy Way,et al.  Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation , 2019, MTSummit.

[11]  Toshiaki Nakazawa,et al.  ASPEC: Asian Scientific Paper Excerpt Corpus , 2016, LREC.

[12]  Rico Sennrich,et al.  Context-Aware Neural Machine Translation Learns Anaphora Resolution , 2018, ACL.

[13]  Jörg Tiedemann,et al.  Neural Machine Translation with Extended Context , 2017, DiscoMT@EMNLP.

[14]  Rico Sennrich,et al.  A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation , 2018, WMT.

[15]  James Henderson,et al.  Document-Level Neural Machine Translation with Hierarchical Attention Networks , 2018, EMNLP.

[16]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[17]  Eiichiro Sumita,et al.  Recycling a Pre-trained BERT Encoder for Neural Machine Translation , 2019, EMNLP.

[18]  Masaaki Nagata,et al.  A Test Set for Discourse Translation from Japanese to English , 2020, LREC.

[19]  Pierre Isabelle,et al.  A Challenge Set Approach to Evaluating Machine Translation , 2017, EMNLP.

[20]  Language independent null subject prediction for statistical machine translation , 2015 .

[21]  Noah A. Smith,et al.  Evaluating Gender Bias in Machine Translation , 2019, ACL.

[22]  Macduff Hughes.  Keynote: Machine Translation Beyond the Sentence , 2018, AMTA.

[23]  Graham Neubig,et al.  Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis , 2011, ACL.

[24]  Andrei Popescu-Belis,et al.  Machine Translation of Spanish Personal and Possessive Pronouns Using Anaphora Probabilities , 2017, EACL.

[25]  Lucia Specia,et al.  Personalized Machine Translation: Preserving Original Author Traits , 2016, EACL.

[26]  I. Willis.  Pronouns , 2004, Encyclopedia of Queer Studies in Education.

[27]  Diyi Yang,et al.  Automatically Neutralizing Subjective Bias in Text , 2019, AAAI.

[28]  Liane Guillou.  Automatic Post-Editing for the DiscoMT Pronoun Translation Task , 2015, DiscoMT@EMNLP.

[29]  Daniel Jurafsky,et al.  JESC: Japanese-English Subtitle Corpus , 2017, LREC.

[30]  Deyi Xiong,et al.  Detecting and Translating Dropped Pronouns in Neural Machine Translation , 2019, NLPCC.

[31]  Gerald Albaum,et al.  The Likert Scale Revisited , 1997 .

[32]  Fabrizio Sebastiani,et al.  An Axiomatically Derived Measure for the Evaluation of Classification Algorithms , 2015, ICTIR.

[33]  Diana Inkpen,et al.  Real-Word Spelling Correction using Google Web 1T 3-grams , 2009, EMNLP.

[34]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[35]  Preslav Nakov,et al.  Findings of the 2017 DiscoMT Shared Task on Cross-lingual Pronoun Prediction , 2017, DiscoMT@EMNLP.

[36]  Katsuhito Sudoh,et al.  Zero Pronoun Resolution can Improve the Quality of J-E Translation , 2012, SSST@ACL.

[37]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[38]  Hiromi Nakaiwa,et al.  Automatic Extraction of Rules for Anaphora Resolution of Japanese Zero Pronouns in Japanese–English Machine Translation from Aligned Sentence Pairs , 1999, Machine Translation.

[39]  Ankur Bapna,et al.  The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation , 2018, ACL.

[40]  Shamil Chollampatt,et al.  Can Automatic Post-Editing Improve NMT? , 2020, EMNLP.

[41]  Antoni Oliver,et al.  Does NMT make a difference when post-editing closely related languages? The case of Spanish-Catalan , 2019, MTSummit.

[42]  Omer Levy,et al.  What Does BERT Look at? An Analysis of BERT’s Attention , 2019, BlackboxNLP@ACL.

[43]  Xing Wang,et al.  One Model to Learn Both: Zero Pronoun Prediction and Translation , 2019, EMNLP.

[44]  Sharid Loáiciga,et al.  Improving machine translation of null subjects in Italian and Spanish , 2012, EACL.

[45]  Daniel Jurafsky,et al.  Towards a Literary Machine Translation: The Role of Referential Cohesion , 2012, CLfL@NAACL-HLT.