Evaluation of the ATR-matrix speech translation system with a pair comparison method between the system and humans

The main goal of the present paper is to propose a new scheme for the overall evaluation of a speech translation system that supports the design of target application systems and determines their performance. Evaluations are conducted on the Japanese-toEnglish ATR-MATRIX speech translation system, which was developed at ATR Interpreting Telecommunications Research Laboratories. In the proposed scheme, the system is compared with native Japanese taking the Test of English for International Communication (TOEIC) for speech translation capability. A regression analysis using evaluation results shows that the speech translation capability of ATR-MATRIX matches Japanese scoring around 500 on the TOEIC.