Automatic Post-Editing for Machine Translation

Automatic Post-Editing (APE) aims to correct systematic errors in a machine translated text. This is primarily useful when the machine translation (MT) system is not accessible for improvement, leaving APE as a viable option to improve translation quality as a downstream task - which is the focus of this thesis. This field has received less attention compared to MT due to several reasons, which include: the limited availability of data to perform a sound research, contrasting views reported by different researchers about the effectiveness of APE, and limited attention from the industry to use APE in current production pipelines. In this thesis, we perform a thorough investigation of APE as a downstream task in order to: i) understand its potential to improve translation quality; ii) advance the core technology - starting from classical methods to recent deep-learning based solutions; iii) cope with limited and sparse data; iv) better leverage multiple input sources; v) mitigate the task-specific problem of over-correction; vi) enhance neural decoding to leverage external knowledge; and vii) establish an online learning framework to handle data diversity in real-time. All the above contributions are discussed across several chapters, and most of them are evaluated in the APE shared task organized each year at the Conference on Machine Translation. Our efforts in improving the technology resulted in the best system at the 2017 APE shared task, and our work on online learning received a distinguished paper award at the Italian Conference on Computational Linguistics. Overall, outcomes and findings of our work have boost interest among researchers and attracted industries to examine this technology to solve real-word problems.

[1]  José Guilherme Camargo de Souza,et al.  FBK-UEdin Participation to the WMT13 Quality Estimation Shared Task , 2013, WMT@ACL.

[2]  Marcin Junczys-Dowmunt,et al.  The AMU System in the CoNLL-2014 Shared Task: Grammatical Error Correction by Data-Intensive and Feature-Rich Statistical Machine Translation , 2014, CoNLL Shared Task.

[3]  Yann Dauphin,et al.  Convolutional Sequence to Sequence Learning , 2017, ICML.

[4]  Christopher Hogan,et al.  Toward the Development of a Post-Editing Module for Machine Translation Raw Output , 2000 .

[5]  Marcin Junczys-Dowmunt,et al.  MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing , 2018, WMT.

[6]  Ramón Fernández Astudillo,et al.  Pushing the Limits of Translation Quality Estimation , 2017, TACL.

[7]  Philipp Koehn,et al.  Findings of the 2017 Conference on Machine Translation (WMT17) , 2017, WMT.

[8]  Philipp Koehn,et al.  Results of the WMT15 Metrics Shared Task , 2015, WMT@EMNLP.

[9]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[10]  Ole Tange,et al.  GNU Parallel: The Command-Line Power Tool , 2011, login Usenix Mag..

[11]  Ramón Fernández Astudillo,et al.  Unbabel's Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task , 2016, WMT.

[12]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[13]  Daniel Ortiz-Martínez Online Learning for Statistical Machine Translation , 2016, Computational Linguistics.

[14]  Jong-Hyeok Lee,et al.  Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation , 2017, WMT.

[15]  Ondrej Bojar,et al.  Results of the WMT17 Metrics Shared Task , 2017, WMT.

[16]  Georges Linarès,et al.  Statistical Post-Editing of Machine Translation for Domain Adaptation , 2012, EAMT.

[17]  Rico Sennrich,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[18]  Mauro Cettolo,et al.  Online Learning Approaches in Computer Assisted Translation , 2013, WMT@ACL.

[19]  Alex Waibel,et al.  Adaptation of the translation model for statistical machine translation based on information retrieval , 2005, EAMT.

[20]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[21]  Matthew D. Zeiler ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[22]  Ventsislav Zhechev Machine Translation Infrastructure and Post-editing Performance at Autodesk , 2012, AMTA.

[23]  Philipp Koehn,et al.  Statistical Post Editing and Dictionary Extraction: Systran/Edinburgh Submissions for ACL-WMT2009 , 2009, WMT@EACL.

[24]  Hervé Blanchon,et al.  Towards a better understanding of statistical post-editing , 2012, IWSLT.

[25]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[26]  Olivier Pietquin,et al.  LIG-CRIStAL Submission for the WMT 2017 Automatic Post-Editing Task , 2017, WMT.

[27]  Rico Sennrich,et al.  Edinburgh Neural Machine Translation Systems for WMT 16 , 2016, WMT.

[28]  Nicola Ueffing,et al.  Generating titles for millions of browse pages on an e-Commerce site , 2017, INLG.

[29]  Roland Kuhn,et al.  Rule-Based Translation with Statistical Phrase-Based Post-Editing , 2007, WMT@ACL.

[30]  Marco Turchi,et al.  Online Automatic Post-Editing across Domains , 2016, CLiC-it/EVALITA.

[31]  Karin M. Verspoor,et al.  Findings of the 2016 Conference on Machine Translation , 2016, WMT.

[32]  D. Hardt,et al.  Incremental Re-training for Post-editing SMT , 2010, AMTA.

[33]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[34]  Yoshua Bengio,et al.  On Using Very Large Target Vocabulary for Neural Machine Translation , 2014, ACL.

[35]  John DeNero,et al.  Hierarchical Incremental Adaptation for Statistical Machine Translation , 2015, EMNLP.

[36]  Philipp Koehn,et al.  Findings of the 2014 Workshop on Statistical Machine Translation , 2014, WMT@ACL.

[37]  Qun Liu,et al.  A Novel Rule Refinement Method for SMT through Simulated Post-Editing , 2014, NLPCC.

[38]  Philipp Koehn,et al.  Margin Infused Relaxed Algorithm for Moses , 2011, Prague Bull. Math. Linguistics.

[39]  Gorka Labaka,et al.  Statistical Post-Editing : A Valuable Method in Domain Adaptation of RBMT Systems for Less-Resourced Languages , 2008 .

[40]  Marion Weller,et al.  Exploring the Planet of the APEs: a Comparative Study of State-of-the-art Methods for MT Automatic Post-Editing , 2015, ACL.

[41]  George F. Foster,et al.  PEPr: Post-Edit Propagation Using Phrase-based Statistical Machine Translation , 2013, MTSUMMIT.

[42]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[43]  Mihael Arcan,et al.  Instance Selection for Online Automatic Post-Editing in a multi-domain scenario , 2016, AMTA.

[44]  Mauro Cettolo,et al.  IRSTLM: an open source toolkit for handling large scale language models , 2008, INTERSPEECH.

[45]  Aapo Hyvärinen,et al.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models , 2010, AISTATS.

[46]  Philipp Koehn,et al.  Findings of the 2018 Conference on Machine Translation (WMT18) , 2018, WMT.

[47]  Helmut Schmid,et al.  Improvements in Part-of-Speech Tagging with an Application to German , 1999 .

[48]  Kevin Knight,et al.  Automated Postediting of Documents , 1994, AAAI.

[49]  Germán Sanchis-Trilles,et al.  Integrating online and active learning in a computer-assisted translation workbench , 2014, AMTA.

[50]  Stefan Riezler,et al.  QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation , 2015, WMT@EMNLP.

[51]  Ulrich Germann Dynamic Phrase Tables for Machine Translation in an Interactive Post-editing Scenario , 2014 .

[52]  Philipp Koehn,et al.  Factored Translation Models , 2007, EMNLP.

[53]  Bill Byrne,et al.  Syntactically Guided Neural Machine Translation , 2016, ACL.

[54]  Josef van Genabith,et al.  Neural Automatic Post-Editing Using Prior Alignment and Reranking , 2017, EACL.

[55]  Francisco Casacuberta,et al.  Translating without in-domain corpus: Machine translation post-editing with online learning techniques , 2015, Comput. Speech Lang..

[56]  Chris Hokamp,et al.  Ensembling Factored Neural Machine Translation Models for Automatic Post-Editing and Quality Estimation , 2017, WMT.

[57]  Matteo Negri,et al.  FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task , 2014, WMT@ACL.

[58]  Alon Lavie,et al.  Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability , 2011, ACL.

[59]  Franz Josef Och,et al.  Minimum Error Rate Training in Statistical Machine Translation , 2003, ACL.

[60]  Francisco Casacuberta,et al.  The New Thot Toolkit for Fully-Automatic and Interactive Statistical Machine Translation , 2014, EACL.

[61]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[62]  Mauro Cettolo,et al.  The repetition rate of text as a predictor of the effectiveness of machine translation adaptation , 2014, AMTA.

[63]  Matthew G. Snover,et al.  A Study of Translation Edit Rate with Targeted Human Annotation , 2006, AMTA.

[64]  Qun Liu,et al.  Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search , 2017, ACL.

[65]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[66]  Alex Kulesza,et al.  Confidence Estimation for Machine Translation , 2004, COLING.

[67]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[68]  José Guilherme Camargo de Souza,et al.  Adaptive Quality Estimation for Machine Translation , 2014, ACL.

[69]  Timothy Baldwin,et al.  Continuous Measurement Scales in Human Evaluation of Machine Translation , 2013, LAW@ACL.

[70]  Timothy Baldwin,et al.  Can machine translation systems be evaluated by the crowd alone , 2015, Natural Language Engineering.

[71]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[72]  Jan Niehues,et al.  Pre-Translation for Neural Machine Translation , 2016, COLING.

[73]  Mattia Antonino Di Gangi,et al.  FBK’s Neural Machine Translation Systems for IWSLT 2016 , 2016, IWSLT.

[74]  Ashish Vaswani,et al.  Decoding with Large-Scale Neural Language Models Improves Translation , 2013, EMNLP.

[75]  Santanu Pal,et al.  Multi-source Neural Automatic Post-Editing: FBK’s participation in the WMT 2017 APE shared task , 2017, WMT.

[76]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[77]  Bo Wang,et al.  SYSTRAN's Pure Neural Machine Translation Systems , 2016, ArXiv.

[78]  Satoshi Nakamura,et al.  Incorporating Discrete Translation Lexicons into Neural Machine Translation , 2016, EMNLP.

[79]  Marco Turchi,et al.  Online Automatic Post-editing for MT in a Multi-Domain Translation Environment , 2017, EACL.

[80]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[81]  Timothy Baldwin,et al.  Accurate Evaluation of Segment-level Machine Translation Metrics , 2015, NAACL.

[82]  Kathleen McKeown,et al.  Learning to Automatically Post-Edit Dropped Words in MT , 2012, AMTA.

[83]  Jörg Tiedemann,et al.  Parallel Data, Tools and Interfaces in OPUS , 2012, LREC.

[84]  Kenneth Heafield,et al.  KenLM: Faster and Smaller Language Model Queries , 2011, WMT@EMNLP.

[85]  Lucia Specia,et al.  Combining Quality Estimation and Automatic Post-editing to Enhance Machine Translation output , 2018, AMTA.

[86]  Marcin Junczys-Dowmunt,et al.  Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing , 2016, WMT.

[87]  Andreas Eisele,et al.  DGT-TM: A freely available Translation Memory in 22 languages , 2012, LREC.

[88]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[89]  Marcello Federico,et al.  Match without a Referee: Evaluating MT Adequacy without Reference Translations , 2012, WMT@NAACL-HLT.

[90]  Benjamin Lecouteux,et al.  LIG System for WMT13 QE Task: Investigating the Usefulness of Features in Word Confidence Estimation for MT , 2013, WMT@ACL.

[91]  Marco Turchi,et al.  Multi-source transformer with combined losses for automatic post editing , 2018, WMT.

[92]  Elisa Ricci,et al.  Online Multitask Learning for Machine Translation Quality Estimation , 2015, ACL.

[93]  Michel Simard,et al.  Statistical Phrase-Based Post-Editing , 2007, NAACL.

[94]  Francisco Casacuberta,et al.  Cost-sensitive active learning for computer-assisted translation , 2014, Pattern Recognit. Lett..

[95]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[96]  Christopher D. Manning,et al.  Stanford Neural Machine Translation Systems for Spoken Language Domains , 2015, IWSLT.

[97]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[98]  Hermann Ney,et al.  CharacTer: Translation Edit Rate on Character Level , 2016, WMT.

[99]  Philipp Koehn,et al.  Findings of the 2013 Workshop on Statistical Machine Translation , 2013, WMT@ACL.

[100]  Francisco Casacuberta,et al.  Statistical Post-Editing of a Rule-Based Machine Translation System , 2009, NAACL.

[101]  Philipp Koehn,et al.  Findings of the 2015 Workshop on Statistical Machine Translation , 2015, WMT@EMNLP.

[102]  Philipp Koehn,et al.  Findings of the 2012 Workshop on Statistical Machine Translation , 2012, WMT@NAACL-HLT.

[103]  Marco Turchi,et al.  The FBK Participation in the WMT15 Automatic Post-editing Shared Task , 2015 .

[104]  Zoubin Ghahramani,et al.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks , 2015, NIPS.

[105]  Lucia Specia,et al.  Guiding Neural Machine Translation Decoding with External Knowledge , 2017, WMT.

[106]  Yifan He,et al.  An Evaluation of Statistical Post-Editing Systems Applied to RBMT and SMT Systems , 2012, COLING.

[107]  EHARA Terumasa,et al.  Rule based machine translation combined with statistical post editor for Japanese to English patent translation , 2007, MTSUMMIT.

[108]  Tomaz Erjavec,et al.  The JRC-Acquis: A Multilingual Aligned Parallel Corpus with 20+ Languages , 2006, LREC.

[109]  Christoph Goller,et al.  Learning task-dependent distributed representations by backpropagation through structure , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[110]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[111]  Francisco Casacuberta,et al.  Active Learning for Interactive Neural Machine Translation of Data Streams , 2018, CoNLL.

[112]  Philip Koehn,et al.  Statistical Machine Translation , 2010, EAMT.

[113]  Jakob Elming,et al.  Transformation-Based Correction of Rule-Based MT , 2006, EAMT.

[114]  Y. Singer,et al.  Ultraconservative online algorithms for multiclass problems , 2003 .

[115]  Mauro Cettolo,et al.  Cache-based Online Adaptation for Machine Translation Enhanced Computer Assisted Translation , 2013, MTSUMMIT.

[116]  David Grangier,et al.  QuickEdit: Editing Text & Translations via Simple Delete Actions , 2017, ArXiv.

[117]  Rico Sennrich,et al.  Linguistic Input Features Improve Neural Machine Translation , 2016, WMT.

[118]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[119]  Josef van Genabith,et al.  A Neural Network based Approach to Automatic Post-Editing , 2016, ACL.

[120]  Lucia Specia,et al.  Machine translation evaluation versus quality estimation , 2010, Machine Translation.

[121]  Zhiming Chen,et al.  Neural Post-Editing Based on Quality Estimation , 2017, WMT.

[122]  W. E. Johnson I.—PROBABILITY: THE DEDUCTIVE AND INDUCTIVE PROBLEMS , 1932 .

[123]  Joel D. Martin,et al.  Improving Translation Quality by Discarding Most of the Phrasetable , 2007, EMNLP.

[124]  Marcin Junczys-Dowmunt,et al.  The AMU-UEdin Submission to the WMT 2017 Shared Task on Automatic Post-Editing , 2017, WMT.

[125]  Ondrej Bojar,et al.  CUNI System for WMT17 Automatic Post-Editing Task , 2017, WMT.

[126]  Lucia Specia,et al.  QuEst - A translation quality estimation framework , 2013, ACL.

[127]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[128]  Lucia Specia,et al.  Findings of the WMT 2018 Shared Task on Quality Estimation , 2018, WMT.

[129]  Josef van Genabith,et al.  Statistical Post-Editing for a Statistical MT System , 2011, MTSUMMIT.

[130]  Jindřich Helcl,et al.  CUNI System for WMT16 Automatic Post-Editing and Multimodal Translation Tasks , 2016, WMT.

[131]  Matteo Negri,et al.  The FBK Participation in the WMT 2016 Automatic Post-editing Shared Task , 2015, WMT.

[132]  Philipp Koehn,et al.  Europarl: A Parallel Corpus for Statistical Machine Translation , 2005, MTSUMMIT.

[133]  Philipp Koehn,et al.  Statistical Significance Tests for Machine Translation Evaluation , 2004, EMNLP.

[134]  John S. White,et al.  The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future Approaches , 1994, AMTA.

[135]  Stephan Vogel,et al.  Parallel Implementations of Word Alignment Tool , 2008, SETQALNLP.

[136]  Alon Lavie,et al.  Learning from Post-Editing: Online Model Adaptation for Statistical Machine Translation , 2014, EACL.

[137]  Philipp Koehn,et al.  Statistical Post-Editing on SYSTRAN‘s Rule-Based Translation System , 2007, WMT@ACL.

[138]  Josef van Genabith,et al.  USAAR: An Operation Sequential Model for Automatic Statistical Post-Editing , 2016, WMT.

[139]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[140]  Hanna Bechara,et al.  Statistical post-editing and quality estimation for machine translation systems , 2013 .

[141]  Abdol Hamid Pilevar,et al.  USING STATISTICAL POST-EDITING TO IMPROVE THE OUTPUT OF RULE-BASED MACHINE TRANSLATION SYSTEM , 2011 .

[142]  Anton Frolov,et al.  YSDA Participation in the WMT’16 Quality Estimation Shared Task , 2016, WMT.

[143]  George F. Foster,et al.  Batch Tuning Strategies for Statistical Machine Translation , 2012, NAACL.

[144]  Mauro Cettolo,et al.  On the Evaluation of Adaptive Machine Translation for Human Post-Editing , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.