Precise measurement method of a speech translation system’s capability with a paired comparison method between the system and humans

Precise Measurement Method of a Speech Translation System’s Capability with a Paired Comparison Method between the System and Humans Fumiaki Sugaya1, Keiji Yasuda1,2, Toshiyuki Takezawa 1, Seiichi Yamamoto1 1ATR Spoken Language Translation Research Laboratories 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan Email: sugaya@slt.atr.co.jp 2Graduate School of Engineering, Doshisha University 1-3, Tatara-miyakodani, Kyotanabe, Kyoto 610-0394, Japan Abstract The main goal of the present paper is to propose new schemes for the overall evaluation of a speech translation system. These schemes are expected to support and improve the design of the target application system, and precisely determine its performance. Experiments are conducted on the Japanese-to-English speech translation system ATR-MATRIX, which was developed at ATR Interpreting Telecommunications Research Laboratories. In the proposed schemes, the system’s translations are compared with those of a native Japanese taking the Test of English for International Communication (TOEIC), which is used as a measure of one’s speech translation capability. Subjective and automatic comparisons are made and the results are compared. A regression analysis on the subjective results shows that the speech translation capability of ATR-MATRIX matches a Japanese person scoring around 500 on the TOEIC. The automatic comparisons also show promising results. Keywords ATR-MATRIX, speech translation system, evaluation, paired comparison, TOEIC 1. Introduction ATR Interpreting Telecommunications Research Laboratories earlier developed the ATR-MATRIX speech translation system (Takezawa et al., 1998), which translates both ways between English and Japanese. At ATR-SLT, we have been carrying out overall evaluations of this system through dialog tests and analyses (Sugaya et al., 1999). We have shown the effectiveness of the system in the basic hotel reservation task/domain. Dialog tests are effective for evaluating the system. However, they do have demerits too, e.g., a lot of labor is required like test control, transcription, and tagging. The Verbmobil project (Wahlster, 2000) conducted end-to-end evaluations to analyze the Verbmobil system. From our experiences, however, it is difficult to enlarge the evaluation target domains/tasks in the same way for ATR-MATRIX. Additional measures would be necessary to support the design of the system to meet performance expectations. Machine translation systems have been evaluated with A, B, C, and D ranks (Sumita et al., 1999). This rank evaluation approach is useful for making relative system comparisons in time series among several schemes. However, one of its demerits is the lack of a direct relationship with the objective performance levels of the real target application systems. Tomita (Tomita et al., 1993) proposed a new scheme using the Test of English as a Foreign Language (TOEFL) to evaluate the quality of translated text as a whole. Evaluation results with this scheme can support the design of the target application systems and determine their performance. The scheme, however, cannot be applied to present speech translation systems, because its task/domain is limited. We propose a new method (Sugaya et al., 2000) that is applicable to speech translation systems with a limited task/domain capability. In this method, both the system and humans with variable translation capabilities answer questions on the translations of test utterances …

Content truncated to 500 words