Applications of Automatic Evaluation Methods to Measuring a Capability of Speech Translation System

Applications of Automatic Evaluation Methods to Measuring a Capability of Speech Translation System Keiji Yasuda ATR Spoken Language Translation Research Laboratories 2-2-2, Hikaridai,"Keihanna Science City", Kyoto, 619-0288, Japan keiji . yasuda@atr . co . jp Also at Graduate School of Engineering, Doshisha University Fumiaki Sugaya KDDI R&D Laboratorie 2-1-15, Ohara, Kamifukuoka-city, Saitama, 356-8502, Japan fsugaya@kddilabs.jp Toshiyuki Takezawa ATR Spoken Language Translation Research Laboratories 2-2-2, Hikaridai, "Keihanna Science City", Kyoto, 619-0288, Japan Seiichi Yamamoto� ATR Spoken Language�Masuzo Yanagida Translation Research Laboratories�Doshisha University 1 -3, Tatara-miyakodani, Kyotanabe, 2-2-2, Hikaridai, "Keihanna Science City",� Kyoto, 619 -0288, Japan�Kyoto, 610-0394, Japan myanagid@mail.doshisha.ac.jp seiichi.yamamoto@atr.co.jp Abstract The main goal of this paper is to pro- pose automatic schemes for the trans- lation paired comparison method. This method was proposed to precisely eval- uate a speech translation system's capa- bility. Furthermore, the method gives an objective evaluation result, i.e., a score of the Test of English for International Communication (TOEIC). The TOEIC score is used as a measure of one's speech translation capability. However, this method requires tremendous eval- uation costs. Accordingly, automatiza- tion of this method is an important sub- ject for study. In the proposed method, currently available automatic evaluation methods are applied to automate the translation paired comparison method. In the experiments, several automatic evaluation methods (BLEU, NIST, DP- based method) are applied. The exper- imental results of these automatic mea- sures show a good correlation with eval- uation results of the translation paired comparison method. 1 Introduction ATR Interpreting Telecommunications Research Laboratories (ATR-ITL) developed the ATR- MATRIX (ATR's Multilingual Automatic Trans- lation System for Information Exchange) speech translation system (Takezawa et al., 1998), which translates both ways between English and Japanese. ATR-ITL has also been carrying out comprehensive evaluations of this system through dialog tests and analyses and has shown the effec- tiveness of the system for basic travel conversa- tion (Sugaya et al., 1999). These experiences, however, indicated that it would be difficult to enlarge the evaluation target domain/task by simply adopting the dialog tests which is employed in the same way for ATR- MATRIX. Additional measures would be neces- 371 ,pane, Text Thm.latIon Ee,ult by Human EvaMM. Sheet Paired Compm.is M A almk Text Japanese Recogmbm ,k1p1m, Japan, e -to .English Lanpunt 'Laudation (J-E IDMI) Figure 1: Diagram of translation paired compari- son method sary in the design of an expanded system in order to meet performance expectations. Sugaya et al. (2000) proposed the translation paired comparison method, which is applicable to precise evaluation of speech translation systems with a limited task/domain capability. A major disadvantage of the translation paired comparison method is its subjective approach to evaluation. Such an approach requires large costs and a long evaluation time. Therefore, automatization of this method remains an important issue to solve. Several automatic evaluation methods have been proposed to achieve efficient development of MT technology, (Su et al., 1992; Papineni et al., 2002; NIST, 2002). Both subjective and automatic evaluation methods are useful for making compar- isons among different schemes or systems. How- …

Content truncated to 500 words