A preliminary evaluation of metadata records machine translation

Purpose – The purpose of this study is to evaluate freely available machine translation (MT) services' performance in translating metadata records.Design/methodology/approach – Randomly selected metadata records were translated from English into Chinese using Google, Bing, and SYSTRAN MT systems. These translations were then evaluated using a five point scale for both fluency and adequacy. Missing count (words not translated) and incorrect count (words incorrectly translated) were also recorded.Findings – Concerning both fluency and adequacy, Google and Bing's translations of more than 70 percent of test data received scores equal to or greater than three, representative of “non‐native Chinese” and “much coverage,” respectively. SYSTRAN scored lowest in both measures. However, these differences were not statistically significant. A Pearson correlation analysis demonstrated a strong relationship (r=0.86) between fluency and adequacy. Missing count and incorrect count strongly correlated with fluency and ad...

[1]  Alon Lavie,et al.  Multi-engine machine translation guided by explicit word matching , 2005, EAMT.

[2]  C. Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science) , 2005 .

[3]  Philipp Koehn,et al.  Further Meta-Evaluation of Machine Translation , 2008, WMT@ACL.

[4]  Philipp Koehn,et al.  (Meta-) Evaluation of Machine Translation , 2007, WMT@ACL.

[5]  Allison Druin,et al.  The International Children’s Digital Library: A Case Study in Designing for a Multilingual, Multicultural, Multigenerational Audience , 2005 .

[6]  Jianqiang Wang,et al.  User-assisted query translation for interactive cross-language information retrieval , 2008, Inf. Process. Manag..

[7]  Jiangping Chen,et al.  Information access across languages on the web: From search engines to digital libraries , 2009, ASIST.

[8]  Carol Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems , 2003, Lecture Notes in Computer Science.

[9]  Klaus Krippendorff,et al.  Computing Krippendorff's Alpha-Reliability , 2011 .

[10]  Hermann Ney,et al.  Human Evaluation of Machine Translation Through Binary System Comparisons , 2007, WMT@ACL.

[11]  Sergei Nirenburg,et al.  Toward Multi-Engine Machine Translation , 1994, HLT.

[12]  Jiangping Chen,et al.  A lexical knowledge base approach for English-Chinese cross-language information retrieval , 2006, J. Assoc. Inf. Sci. Technol..

[13]  Hermann Ney,et al.  Computing Consensus Translation for Multiple Machine Translation Systems Using Enhanced Hypothesis Alignment , 2006, EACL.

[14]  Christine L. Borgman,et al.  What are Digital Libraries? Competing Visions , 1999, Inf. Process. Manag..

[15]  Dan Wu,et al.  Exploring the Further Integration of Machine Translation in Multilingual Information Access , 2010 .

[16]  Greg R. Notess,et al.  Multilingual Searching : Search Engine Language Tools , 2008 .

[17]  Chris Callison-Burch,et al.  Proceedings of the Second Workshop on Statistical Machine Translation , 2007 .

[18]  Taro Watanabe,et al.  Using Language and Translation Models to Select the Best among Outputs from Multiple MT Systems , 2002, COLING.

[19]  Suresh Jange,et al.  Digital libraries : an overview , 2001 .

[20]  Tadashi Nomoto.  Multi-Engine Machine Translation with Voted Language Model , 2004, ACL.

[21]  William Y. Arms.  Automated Digital Libraries: How Effectively Can Computers Be Used for the Skilled Tasks of Professional Librarianship? , 2000, D Lib Mag..

[22]  Chris Callison-Burch,et al.  A program for automatically selecting the best output from multiple machine translation engines , 2001, MTSUMMIT.

[23]  Sarah E Yates.  Scaling the tower of babel fish: An analysis of the machine translation of legal information , 2006 .

[24]  Carol Peters,et al.  Comparative evaluation of multilingual information access systems : 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August 21-22, 2003 : revised papers , 2004 .

[25]  Mark A. Przybocki,et al.  Translation Adequacy and Preference Evaluation Tool (TAP-ET) , 2008, LREC.

[26]  Uwe Küssner,et al.  Learning to Select a Good Translation , 2000, COLING.

[27]  Giuseppe Riccardi,et al.  Computing consensus translation from multiple machine translation systems , 2001, IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU '01..

[28]  Andreas Eisele,et al.  Multi-engine machine translation with an open-source decoder for statistical machine translation , 2007 .

[29]  Benjamin B. Bederson,et al.  Translation by iterative collaboration between monolingual users , 2010, HCOMP '10.

[30]  Richard M. Schwartz,et al.  Combining Outputs from Multiple Machine Translation Systems , 2007, NAACL.

[31]  Jiangping Chen,et al.  Cross-language Search: The Case of Google Language Tools , 2009, First Monday.

[32]  Alon Lavie,et al.  The significance of recall in automatic metrics for MT evaluation , 2004, AMTA.