An automatic evaluation method of translation quality using translation answer candidates queried from a parallel corpus

An Automatic Evaluation Method of Translation Quality Using Translation Answer Candidates Queried from a Parallel Corpus Keiji Yasuda1, 2, Fumiaki Sugaya1, Toshiyuki Takezawa1, Seiichi Yamamoto1, Masuzo Yanagida2 1ATR Spoken Language Translation Research Laboratories 2-2-2, Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0288 JAPAN keiji.yasuda@slt.atr.co.jp 2Graduate School of Engineering, Doshisha University 1-3, Tatara-miyakodani, Kyotanabe, Kyoto, 610-0394 JAPAN Abstract An automatic translation quality evaluation method is proposed. In the proposed method, a parallel corpus is used to query translation answer candidates. The translation output is evaluated by measuring the similarity between the translation output and translation answer candidates with DP matching. This method evaluates a language translation subsystem of the Japanese-to-English ATR- MATRIX speech translation system developed at ATR Interpreting Telecommunications Research Laboratories. Discriminant analysis is then carried out to examine the evaluation performance of the proposed method. Experimental results show the effectiveness of the proposed method. The discriminant ratio is 83.5% for 2-class discrimination between absolutely correct and less appropriate translations classified subjectively. Also discussed are issues of the proposed method when it is applied to speech translation systems which inevitably make recognition errors. Keywords MT evaluation techniques, Translation quality, Parallel Corpus, DP matching, ATR-MATRIX 1. Introduction An automatic translation quality evaluation method is required since subjective evaluations require large costs and the turn-around time is very long. Over the past few years, we have carried out several studies on the evaluation of the speech translation system called ATR-MATRIX (ATR’s Multilingual Automatic Translation System for Information Exchange) (Takezawa et al., 1998). Examples of these studies include a rank evaluation method (Sumita et al., 1999), a translation paired comparison method (Sugaya et al., 2000), and an evaluation method using dynamic programming (DP) (Takezawa et al., 1999). The first two methods mentioned above can be considered subjective evaluation methods. The last one is an automatic evaluation method that evaluates the robustness of language translation subsystems against speech recognition errors. This is not to evaluate the translation quality. Keh-Yin Su et al. proposed the most conventional automatic evaluation method of translation quality using DP matching (1992). However, the method appears to lack an evaluation capability to deal with multiple expressions in a target language utterance, since only one translation answer is provided in this method. The method proposed in this paper solves the problem by using multiple translation answer candidates queried from a parallel corpus by DP matching. In section 2, the proposed method is explained. In section 3, evaluation results by the proposed method and the most conventional automatic evaluation method are compared from the viewpoint of evaluation performance. To reduce the subjective rank evaluation cost, and improve the evaluation performance, a hybrid method is explained, that is, a combination between the proposed method and the subjective rank evaluation method. Also discussed are issues of the proposed method when it is applied to evaluation of the outputs of speech translation systems, which inevitably make errors in the recognition stage. In section 4, we state our conclusion. 2. Evaluation Method The similarity between a translation output and …

Content truncated to 500 words