Approaches to Human and Machine Translation Quality Assessment

In both research and practice, translation quality assessment is a complex task involving a range of linguistic and extra-linguistic factors. This chapter provides a critical overview of the established and developing approaches to the definition and measurement of translation quality in human and machine translation workflows across a range of research, educational, and industry scenarios. We intertwine literature from several interrelated disciplines dealing with contemporary translation quality assessment and, while we acknowledge the need for diversity in these approaches, we argue that there are fundamental and widespread issues that remain to be addressed, if we are to consolidate our knowledge and practice of translation quality assessment in increasingly technologised environments across research, teaching, and professional practice.

[1]  Jörg Tiedemann,et al.  Statistical Machine Translation with Readability Constraints , 2013, NODALIDA.

[2]  Jody Byrne,et al.  Technical Translation: Usability Strategies for Translating Technical Documentation , 2006 .

[3]  Doug Arnold,et al.  Machine Translation: An Introductory Guide , 1994 .

[4]  Dominic Stewart.  From Pro Loco to Pro Globo , 2013 .

[5]  Germán Sanchis-Trilles,et al.  CASMACAT: An Open Source Workbench for Advanced Computer Aided Translation , 2013, Prague Bull. Math. Linguistics.

[6]  Jenny Williams.  Theories of Translation , 2013 .

[7]  Philip Koehn,et al.  Statistical Machine Translation , 2010, EAMT.

[8]  Preslav Nakov,et al.  Using Discourse Structure Improves Machine Translation Evaluation , 2014, ACL.

[9]  Heiko Sacher,et al.  Beyond Translation: Approaches to Interactive Products for Chinese Consumers , 2001, Int. J. Hum. Comput. Interact..

[10]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments , 2007, WMT@ACL.

[11]  James S. Holmes.  Translated!: Papers on Literary Translation and Translation Studies. With an introduction by Raymond van den Broeck , 1988 .

[12]  Maarit Koponen,et al.  Comparing human perceptions of post-editing effort with post-editing operations , 2012, WMT@NAACL-HLT.

[13]  Joanna Drugan,et al.  Quality In Professional Translation: Assessment and Improvement , 2013 .

[14]  Lucia Specia,et al.  Exploiting Objective Annotations for Minimising Translation Post-editing Effort , 2011, EAMT.

[15]  Sharon O'Brien,et al.  Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking , 2014, Int. J. Hum. Comput. Interact..

[16]  Stuart Campbell,et al.  Translation into the Second Language , 1998 .

[17]  Anthony Pym,et al.  Exploring Translation Theories , 2009 .

[18]  Mona Baker,et al.  In Other Words: A Coursebook on Translation , 1993 .

[19]  Deborah A. Coughlin,et al.  Correlating automated and human assessments of machine translation quality , 2003, MTSUMMIT.

[20]  Tiina Tuominen,et al.  User-Centered Translation , 2014 .

[21]  François Masselot,et al.  A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localisation Context , 2010, Prague Bull. Math. Linguistics.

[22]  Hwee Tou Ng,et al.  Better Evaluation Metrics Lead to Better Machine Translation , 2011, EMNLP.

[23]  Philipp Koehn,et al.  Findings of the 2009 Workshop on Statistical Machine Translation , 2009, WMT@EACL.

[24]  Paul Kussmaul,et al.  Training the Translator , 1995 .

[25]  Harold L. Somers,et al.  Evaluating Machine Translation: the Cloze Procedure Revisited , 2000, TC.

[26]  Jeremy Munday,et al.  Introducing Translation Studies: Theories and Applications , 1991 .

[27]  Joss Moorkens,et al.  Under pressure: translation in times of austerity , 2017 .

[28]  Richard Taffler,et al.  Readability and Understandability: Different Measures of the Textual Complexity of Accounting Narrative , 1992 .

[29]  Ralph Weischedel,et al.  A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION , 2005 .

[30]  Sharon O'Brien,et al.  Correlations of perceived post-editing effort with measurements of actual effort , 2015, Machine Translation.

[31]  Gorka Labaka,et al.  A hybrid machine translation architecture guided by syntax , 2014, Machine Translation.

[32]  Lluís Màrquez i Villodre,et al.  A Smorgasbord of Features for Automatic MT Evaluation , 2008, WMT@ACL.

[33]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[34]  J. House.  Translation quality assessment: A model revisited , 1997 .

[35]  Wioleta Karwacka,et al.  Quality assurance in medical translation , 2014 .

[36]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[37]  B. Harris,et al.  Toward a Science of Translation , 1977 .

[38]  Hala Almaghout,et al.  A survey of machine translation competences: Insights for translation technology educators and practitioners , 2015 .

[39]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[40]  Philipp Koehn,et al.  Enabling Monolingual Translators: Post-Editing vs. Options , 2010, NAACL.

[41]  Federico Gaspari.  Online MT Services and Real Users? Needs: An Empirical Usability Evaluation , 2004, AMTA.

[42]  Niels Ole Bernsen,et al.  Evaluation and usability of multimodal spoken language dialogue systems , 2004, Speech Commun..

[43]  R. Beaugrande,et al.  Introduction to text linguistics , 1981 .

[44]  Ana Guerberof Arenas.  Correlations between productivity and quality when post-editing in a professional context , 2014, Machine Translation.

[45]  Sigrid Klerke,et al.  Reading metrics for estimating task efficiency with MT output , 2015, EMNLP 2015.

[46]  Hans P. Krings,et al.  Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes , 2001 .

[47]  Scott Kushner,et al.  The freelance translation machine: Algorithmic culture and the invisible industry , 2013, New Media Soc..

[48]  Hermann Ney,et al.  Syntax-Oriented Evaluation Measures for Machine Translation Output , 2009, WMT@EACL.

[49]  Katharina Reiss,et al.  Grundlegung einer allgemeinen Translationstheorie , 1984 .

[50]  Andrei Popescu-Belis,et al.  Principles of Context-Based Machine Translation Evaluation , 2002, Machine Translation.

[51]  Inger Lassen,et al.  Accessibility and Acceptability in Technical Manuals: A Survey of Style and Grammatical Metaphor , 2003 .

[52]  Sharon O'Brien.  Post-editing of Machine Translation: Processes and Applications , 2014 .

[53]  Andy Way,et al.  Evaluating machine translation with LFG dependencies , 2007, Machine Translation.

[54]  Colin Harrison,et al.  Readability in the Classroom , 1980 .

[55]  Lucia Specia,et al.  Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles , 2011, RANLP.

[56]  Jun Tang,et al.  Translating into English as a Non-Native Language: a translator trainer’s perspective , 2017 .

[57]  Robert J. Hartsuiker,et al.  The impact of machine translation error types on post-editing effort indicators , 2015, MTSUMMIT.

[58]  Christian Federmann,et al.  Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output , 2012, Prague Bull. Math. Linguistics.

[59]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[60]  Joseph P. Turian,et al.  Evaluation of machine translation and its evaluation , 2003, MTSUMMIT.

[61]  Christina Schäffner,et al.  From 'Good' to 'Functionally Appropriate': assessing translation quality , 1997 .

[62]  Anthony Pym,et al.  Translating as risk management , 2015 .

[63]  Susanne Lauscher,et al.  Translation Quality Assessment , 2000 .

[64]  J. House.  Translation Quality Assessment: Linguistic Description versus Social Evaluation , 1977 .

[65]  Sheila C. M. de Sousa,et al.  Measuring acceptability of machine translated enterprise content , 2016 .

[66]  Maja Popovic,et al.  chrF: character n-gram F-score for automatic MT evaluation , 2015, WMT@EMNLP.

[67]  Stephen Doherty,et al.  Translations| The Impact of Translation Technologies on the Process and Product of Translation , 2016 .

[68]  Philipp Koehn,et al.  Findings of the 2011 Workshop on Statistical Machine Translation , 2011, WMT@EMNLP.

[69]  Alan K. Melby,et al.  What Is Quality? A Management Discipline and the Translation Industry Get Acquainted , 2014 .

[70]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[71]  Daniel Jurafsky,et al.  Measuring machine translation quality as semantic equivalence: A metric based on entailment features , 2009, Machine Translation.

[72]  Juliane House,et al.  Translation Quality Assessment: Past and Present , 2014 .

[73]  Nike K. Pokorn.  Challenging the Traditional Axioms: Translation into a non-mother tongue , 2005 .

[74]  Sharon O'Brien,et al.  Towards predicting post-editing productivity , 2011, Machine Translation.

[75]  Gideon Toury,et al.  Descriptive translation studies and beyond , 1995 .

[76]  Gavriel Salvendy,et al.  Content Preparation and Management for Web Design: Eliciting, Structuring, Searching, and Displaying Information , 2002, Int. J. Hum. Comput. Interact..

[77]  Johann Roturier,et al.  An investigation into the impact of controlled English rules on the comprehensibility, usefulness and acceptability of machine-translated technical documentation for French and German users , 2006 .

[78]  George R. Doddington,et al.  Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics , 2002 .

[79]  M. J. Jones,et al.  A Longitudinal Study of the Readability of the Chairman's Narratives in the Corporate Reports of a UK Company , 1988 .

[80]  Philipp Koehn,et al.  (Meta-) Evaluation of Machine Translation , 2007, WMT@ACL.

[81]  Ilse Depraetere,et al.  What counts as useful advice in a university post-editing training context? Report on a case , 2010, EAMT.

[82]  Ondrej Bojar,et al.  A Grain of Salt for the WMT Manual Evaluation , 2011, WMT@EMNLP.

[83]  Victoria Arranz,et al.  Document-Level Automatic MT Evaluation based on Discourse Representations , 2010, WMT@ACL.

[84]  Sheila Castilho,et al.  Does post-editing increase usability? A study with Brazilian Portuguese as target language , 2014, EAMT.

[85]  Ilse Depraetere,et al.  A contrastive analysis of five automated QA tools (QA Distiller 6.5.8, Xbench 2.8, ErrorSpy 5.0, SDLTrados 2007 QA Checker 2.0 and SDLX 2007 SP2 QA Check) , 2012 .