Human Effort and Machine Learnability in Computer Aided Translation

Analyses of computer aided translation typically focus on either frontend interfaces and human effort, or backend translation and machine learnability of corrections. However, this distinction is artificial in practice since the frontend and backend must work in concert. We present the first holistic, quantitative evaluation of these issues by contrasting two assistive modes: postediting and interactive machine translation (MT). We describe a new translator interface, extensive modifications to a phrasebased MT system, and a novel objective function for re-tuning to human corrections. Evaluation with professional bilingual translators shows that post-edit is faster than interactive at the cost of translation quality for French-English and EnglishGerman. However, re-tuning the MT system to interactive output leads to larger, statistically significant reductions in HTER versus re-tuning to post-edit. Analysis shows that tuning directly to HTER results in fine-grained corrections to subsequent machine output.

[1]  Yehoshua Bar-Hillel,et al.  The Present Status of Automatic Translation of Languages , 1960, Adv. Comput..

[2]  George F. Foster,et al.  TransType: a Computer-Aided Translation Typing System , 2000 .

[3]  George F. Foster,et al.  User-Friendly Text Prediction For Translators , 2002, EMNLP.

[4]  Philip Resnik,et al.  Evaluating Translational Correspondence using Annotation Projection , 2002, ACL.

[5]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[6]  Franz Josef Och,et al.  Minimum Error Rate Training in Statistical Machine Translation , 2003, ACL.

[7]  Hermann Ney,et al.  Efficient Search for Interactive Statistical Machine Translation , 2003, EACL.

[8]  Christopher Culy,et al.  The limits of n-gram translation evaluation metrics , 2003, MTSUMMIT.

[9]  Hermann Ney,et al.  The Alignment Template Approach to Statistical Machine Translation , 2004, CL.

[10]  Ralph Weischedel,et al.  A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION , 2005 .

[11]  Stefan Riezler,et al.  On Some Pitfalls in Automatic Evaluation and Significance Testing for MT , 2005, IEEvaluation@ACL.

[12]  Philipp Koehn,et al.  Re-evaluating the Role of Bleu in Machine Translation Research , 2006, EACL.

[13]  Matthew G. Snover,et al.  A Study of Translation Edit Rate with Targeted Human Annotation , 2006, AMTA.

[14]  Ben Taskar,et al.  Alignment by Agreement , 2006, NAACL.

[15]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[16]  David Chiang,et al.  Forest Rescoring: Faster Decoding with Integrated Language Models , 2007, ACL.

[17]  Wolfgang Macherey,et al.  Lattice-based Minimum Error Rate Training for Statistical Machine Translation , 2008, EMNLP.

[18]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[19]  Christopher D. Manning,et al.  A Simple and Effective Hierarchical Phrase Reordering Model , 2008, EMNLP.

[20]  Hermann Ney,et al.  Statistical Approaches to Computer-Assisted Translation , 2009, CL.

[21]  Francisco Casacuberta,et al.  Interactive Machine Translation Based on Partial Statistical Phrase-based Alignments , 2009, RANLP.

[22]  Nitin Madnani,et al.  Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric , 2009, WMT@EACL.

[23]  Philipp Koehn,et al.  A Web-Based Interactive Computer Aided Translation Tool , 2009, ACL.

[24]  Alon Lavie,et al.  Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level , 2010, NAACL.

[25]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[26]  Philipp Koehn,et al.  A process study of computer-aided translation , 2009, Machine Translation.

[27]  Francisco Casacuberta,et al.  Online Learning for Interactive Statistical Machine Translation , 2010, NAACL.

[28]  Chris Callison-Burch,et al.  Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators , 2010, NAACL.

[29]  Christian Federmann,et al.  Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations , 2010, LREC.

[30]  M. Carl.  A computational framework for a cognitive model of human translation processes , 2010, TC.

[31]  Daniel Jurafsky,et al.  The Best Lexical Metric for Phrase-Based Statistical MT System Optimization , 2010, NAACL.

[32]  Thorsten Joachims,et al.  Online Structured Prediction via Coactive Learning , 2012, ICML.

[33]  Pascual Martínez-Gómez,et al.  Online adaptation strategies for statistical machine translation in post-editing scenarios , 2012, Pattern Recognit..

[34]  George F. Foster,et al.  Batch Tuning Strategies for Statistical Machine Translation , 2012, NAACL.

[35]  Francisco Casacuberta,et al.  Advanced computer aided translation with a web-based workbench , 2013, MTSUMMIT.

[36]  Christopher D. Manning,et al.  Fast and Adaptive Online Training of Feature-Rich Translation Models , 2013, ACL.

[37]  Philipp Koehn,et al.  Scalable Modified Kneser-Ney Language Model Estimation , 2013, ACL.

[38]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[39]  Philipp Koehn,et al.  Findings of the 2013 Workshop on Statistical Machine Translation , 2013, WMT@ACL.

[40]  Jeffrey Heer,et al.  The efficacy of human post-editing for language translation , 2013, CHI.

[41]  Marcello Federico,et al.  Generative and Discriminative Methods for Online Adaptation in SMT , 2013, MTSUMMIT.

[42]  Alon Lavie,et al.  Real Time Adaptive Machine Translation for Post-Editing with cdec and TransCenter , 2014, HaCaT@EACL.

[43]  Christopher D. Manning,et al.  An Empirical Comparison of Features and Tuning for Phrase-based Machine Translation , 2014, WMT@ACL.

[44]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[45]  Matt Post,et al.  Efficient Elicitation of Annotations for Human Evaluation of Machine Translation , 2014, WMT@ACL.

[46]  Philipp Koehn,et al.  Findings of the 2014 Workshop on Statistical Machine Translation , 2014, WMT@ACL.

[47]  Francisco Casacuberta,et al.  The New Thot Toolkit for Fully-Automatic and Interactive Statistical Machine Translation , 2014, EACL.

[48]  Alon Lavie,et al.  Learning from Post-Editing: Online Model Adaptation for Statistical Machine Translation , 2014, EACL.

[49]  Jeffrey Heer,et al.  Predictive translation memory: a mixed-initiative system for human language translation , 2014, UIST.

[50]  Christopher D. Manning,et al.  Phrasal: A Toolkit for New Directions in Statistical Machine Translation , 2014, WMT@ACL.

[51]  Marcello Federico,et al.  Online Word Alignment for Online Adaptive Machine Translation , 2014, HaCaT@EACL.