Dependency-based Automatic Enumeration of Semantically Equivalent Word Orders for Evaluating Japanese Translations

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 287–292, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Dependency-based Automatic Enumeration of Semantically Equivalent Word Orders for Evaluating Japanese Translations Hideki Isozaki, Natsume Kouchi Okayama Prefectural University 111 Kuboki, Soja-shi, Okayama, 719-1197, Japan isozaki@cse.oka-pu.ac.jp Tsutomu Hirao NTT Communication Science Laboratories 2-4, Hikaridai, Seika-cho, Sorakugun, Kyoto, 619-0237, Japan hirao.tsutomu@lab.ntt.co.jp Abstract Scrambling is acceptable reordering of verb arguments in languages such as Japanese and German. In automatic eval- uation of translation quality, BLEU is the de facto standard method, but BLEU has only very weak correlation with hu- man judgements in case of Japanese-to- English/English-to-Japanese translations. Therefore, alternative methods, IMPACT and RIBES, were proposed and they have shown much stronger correlation than BLEU. Now, RIBES is widely used in recent papers on Japanese-related transla- tions. RIBES compares word order of MT output with manually translated reference sentences but it does not regard scram- bling at all. In this paper, we present a method to enumerate scrambled sentences from dependency trees of reference sen- tences. Our experiments based on NTCIR Patent MT data show that the method im- proves sentence-level correlation between RIBES and human-judged adequacy. 1 Introduction Statistical Machine Translation has grown with an automatic evaluation method BLEU (Papineni et al., 2002). BLEU measures local word order by n- grams and does not care about global word order. In JE/EJ translations, this insensitivity degrades BLEU’s correlation with human judgements. Therefore, alternative automatic evaluation methods are proposed. Echizen-ya and Araki (2007) proposed IMPACT. Isozaki et al. (2010) presented the idea of RIBES. Hirao et al. (2011) named this method “RIBES” (Rank-based Intu- itive Bilingual Evaluation Score). This version of RIBES was defined as follows: RIBES = NKT × P α Table 1: Meta-evaluation of NTCIR-7 JE task data (Spearman’s ρ, System-level correlation) BLEU METEOR ROUGE-L IMPACT RIBES 0.515 0.490 0.903 0.826 0.947 where NKT (Normalized Kendall’s τ) is defined by (τ + 1)/2. This NKT is used for measur- ing word order similarity between a reference sen- tence and an MT output sentence. Thus, RIBES penalizes difference of global word order. P is precision of unigrams. RIBES is defined for each test sentence and averaged RIBES is used for eval- uating the entire test corpus. talkTable 1 is a table in an IWSLT-2012 invited (http://hltc.cs.ust.hk/iwslt/slides/ Isozaki2012 slides.pdf). METEOR was pro- posed by Banerjee and Lavie (2005). ROUGE-L was proposed by Lin and Och (2004). According to this table, RIBES with α = 0.2 has a very strong correlation (Spearman’s ρ = 0.947) with human-judged adequacy. For each sentence, we use the average of adequacy scores of three judges. Here, we call this average “Adequacy”. We focus on Adequacy because current SMT systems tend to output inadequate sentences. Note that only single reference translations are available for this task although use of multiple references is common for BLEU. RIBES is publicly available from http:// www.kecl.ntt.co.jp/icl/lirg/ribes/ and was used as a standard quality measure in recent NTCIR PatentMT tasks (Goto et …

Content truncated to 500 words