A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English

ATR-MATRIX is a multi-lingual speech-to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel arrangement domain. In this paper, we propose a new evaluation method for speech translation systems. Our current focus is on measuring the robustness of a language translation sub-system, with quick calculation and low cost. Therefore, we calculate the difference between the translation output from transcription texts and the translation output from input speech by a dynamic programming method. We present the first trial experiment of this method applied to our Japanese-to-English speech translation system. We also provide related discussions on such points as error analysis and the relationship between the proposed method and translation quality evaluation manually done by humans.

[1]  Masami Suzuki,et al.  A prototype of a Japanese-Korean realtime speech translation system , 1995, EUROSPEECH.

[2]  Mari Ostendorf,et al.  HMM topology design using maximum likelihood successive state splitting , 1997, Comput. Speech Lang..

[3]  Yoshinori Sagisaka,et al.  Variable-order N-gram generation by word-class splitting and consecutive word grouping , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[4]  Masaaki Nagata,et al.  ATR's speech translation system: ASURA , 1993, EUROSPEECH.

[5]  H. Iida,et al.  Multi-Lingual Spoken Dialog Translation System Using Transfer-Driven Machine Translation , 1997, MTSUMMIT.

[6]  Vassilios Digalakis,et al.  Spoken language translation with MID-90's technology: a case study , 1993, EUROSPEECH.

[7]  Hitoshi Iida,et al.  A Japanese-to-English speech translation system: ATR-MATRIX , 1998, ICSLP.

[8]  Richard Sproat,et al.  A spoken language translator for restricted-domain context-free languages , 1992, Speech Commun..

[9]  Yoshinori Sagisaka,et al.  Spontaneous dialogue speech recognition using cross-word context constrained word graphs , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[10]  Wolfgang Wahlster,et al.  Verbmobil: the combination of deep and shallow processing for spontaneous speech translation , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Yumi Wakita,et al.  Correct parts extraction from speech recognition results using semantic distance calculation, and its application to speech translation , 1997, Workshop On Spoken Language Translation.

[12]  Alon Lavie,et al.  Janus-III: speech-to-speech translation in multiple languages , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[13]  Toshiyuki Takezawa,et al.  Transformation into language processing units by dividing and connecting utterance units , 1999, EUROSPEECH.

[14]  Hitoshi Iida,et al.  A speech and language database for speech translation research , 1994, ICSLP.

[15]  Toshiyuki Takezawa,et al.  End-to-end evaluation in ATR-MATRIX: speech translation system between English and Japanese , 1999, EUROSPEECH.