Human Evaluation of Online Machine Translation Services for English/Russian-Croatian

This paper presents results of human evaluation of machine translated texts for one non closely-related language pair, English-Croatian, and for one closely-related language pair, Russian-Croatian. 400 sentences from the domain of tourist guides were analysed, i.e. 100 sentences for each language pair and for two online machine translation services, Google Translate and Yandex.Translate. Human evaluation is made with regard to the criteria of fluency and adequacy. In order to measure internal consistency, Cronbach’s alpha is calculated. Error analysis is made for several categories: untranslated/omitted words, surplus words, morphological errors/wrong word endings, lexical errors/wrong translations, syntactic errors/wrong word order and punctuation errors. At the end of this paper, conclusions and suggestions for further research are given.

[1]  Thei Zervaki.  Online free translation services , 2002 .

[2]  Lucia Specia,et al.  Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles , 2011, RANLP.

[3]  Chris Quirk,et al.  Training a Sentence-Level Machine Translation Confidence Measure , 2004, LREC.

[4]  Lucia Specia,et al.  Exploiting Objective Annotations for Measuring Translation Post-editing Effort , 2011 .

[5]  Mahesan Niranjan,et al.  Exploitation of Machine Learning Techniques in Modelling Phrase Movements for Machine Translation , 2011, J. Mach. Learn. Res..

[6]  Philipp Koehn,et al.  Statistical Significance Tests for Machine Translation Evaluation , 2004, EMNLP.

[7]  C. Kit,et al.  Comparative Evaluation of Online Machine Translation Systems with Legal Texts , 2008 .

[8]  Sanja Seljan,et al.  BLEU Evaluation of Machine-Translated English-Croatian Legislation , 2012, LREC.

[9]  Franz Josef Och,et al.  Minimum Error Rate Training in Statistical Machine Translation , 2003, ACL.

[10]  Stephen Hampshire,et al.  Translation and the Internet: Evaluating the Quality of Free Online Machine Translators , 2010 .

[11]  Christopher Cieri,et al.  Corpus Support for Machine Translation at LDC , 2006, LREC.

[12]  Yifan He,et al.  Bridging SMT and TM with Translation Recommendation , 2010, ACL.

[13]  Tatiana Gornostay,et al.  LetsMT! - Online Platform for Sharing Training Data and Building User Tailored Machine Translation , 2010, Baltic HLT.

[14]  María Dolores García Santiago,et al.  Automatic web translators as part of a multilingual question-answering (QA) system: translation of questions , 2010 .

[15]  Jeroen Latour.  Evaluating Statistical Machine Translation from English to Dutch , 2004 .

[16]  Radu Soricut,et al.  TrustRank: Inducing Trust in Automatic Translations via Ranking , 2010, ACL.

[17]  Sanja Seljan,et al.  Evaluation of Free Online Machine Translations for Croatian-English and English-Croatian Language Pairs , 2011 .

[18]  Miles Osborne,et al.  Statistical Machine Translation , 2010, Encyclopedia of Machine Learning and Data Mining.