Automatic Evaluation of Translation Quality for Distant Language Pairs

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944–952, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Automatic Evaluation of Translation Quality for Distant Language Pairs Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, Hajime Tsukada NTT Communication Science Laboratories, NTT Corporation 2-4 Hikaridai, Seikacho, Sorakugun, Kyoto, 619-0237, Japan {isozaki,hirao,kevinduh,sudoh,tsukada}@cslab.kecl.ntt.co.jp Abstract Automatic evaluation of Machine Translation (MT) quality is essential to developing high- quality MT systems. Various evaluation met- rics have been proposed, and BLEU is now used as the de facto standard metric. How- ever, when we consider translation between distant language pairs such as Japanese and English, most popular metrics (e.g., BLEU, NIST, PER, and TER) do not work well. It is well known that Japanese and English have completely different word orders, and special care must be paid to word order in transla- tion. Otherwise, translations with wrong word order often lead to misunderstanding and in- comprehensibility. For instance, SMT-based Japanese-to-English translators tend to trans- late ‘A because B’ as ‘B because A.’ Thus, word order is the most important problem for distant language translation. However, conventional evaluation metrics do not sig- nificantly penalize such word order mistakes. Therefore, locally optimizing these metrics leads to inadequate translations. In this pa- per, we propose an automatic evaluation met- ric based on rank correlation coefficients mod- ified with precision. Our meta-evaluation of the NTCIR-7 PATMT JE task data shows that this metric outperforms conventional metrics. 1 Introduction Automatic evaluation of machine translation (MT) quality is essential to developing high-quality ma- chine translation systems because human evaluation is time consuming, expensive, and irreproducible. If we have a perfect automatic evaluation metric, we can tune our translation system for the metric. BLEU (Papineni et al., 2002b; Papineni et al., 2002a) showed high correlation with human judg- ments and is still used as the de facto standard au- tomatic evaluation metric. However, Callison-Burch et al. (2006) argued that the MT community is overly reliant on BLEU by showing examples of poor per- formance. For Japanese-to-English (JE) translation, Echizen-ya et al. (2009) showed that the popular BLEU and NIST do not work well by using the sys- tem outputs of the NTCIR-7 PATMT (patent transla- tion) JE task (Fujii et al., 2008). On the other hand, ROUGE-L (Lin and Hovy, 2003), Word Error Rate (WER), and IMPACT (Echizen-ya and Araki, 2007) worked better. In these studies, Pearson’s correlation coefficient and Spearman’s rank correlation ρ with human eval- uation scores are used to measure how closely an automatic evaluation method correlates with human evaluation. This evaluation of automatic evaluation methods is called meta-evaluation. In human eval- uation, people judge the adequacy and the fluency of each translation. Denoual and Lepage (2005) pointed out that BLEU assumes word boundaries, which is ambigu- ous in Japanese and Chinese. Here, we assume the word boundaries given by ChaSen, one of the standard morphological analyzers (http://chasen- legacy.sourceforge.jp/) following Fujii et al. (2008) In JE translation, most Statistical Machine Trans- lation …

Content truncated to 500 words