Dependency Analysis of Scrambled References for Better Evaluation of Japanese Translation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 450–456, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Dependency Analysis of Scrambled References for Better Evaluation of Japanese Translations Hideki Isozaki and Natsume Kouchi∗ Okayama Prefectural University 111 Kuboki, Soja-shi, Okayama, 719-1197, Japan isozaki@cse.oka-pu.ac.jp Abstract In English-to-Japanese translation, BLEU (Papineni et al., 2002), the de facto standard evaluation metric for machine translation (MT), has very weak corre- lation with human judgments (Goto et al., 2011; Goto et al., 2013). Therefore, RIBES (Isozaki et al., 2010; Hirao et al., 2014) was proposed. RIBES mea- sures similarity of the word order of a machine-translated sentence and that of a corresponding human-translated reference sentence. RIBES has much stronger correlation than BLEU but most Japanese sentences have alternative word orders (scrambling), and one reference sentence is not sufficient for fair evaluation. Isozaki et al. (2014) pro- posed a solution to this problem. This solution generates semantically equiva- lent word orders of reference sentences. Automatically generated word orders are sometimes incomprehensible or mislead- ing, and they introduced a heuristic rule that filters out such bad sentences. How- ever, their rule is too conservative and generated alternative word orders for only 30% of reference sentences. In this paper, we present a rule-free method that uses a dependency parser to check scrambled sentences and generated alternatives for 80% of sentences. The ex- perimental results show that our method improves sentence-level correlation with human judgments. In addition, strong system-level correlation of single refer- ence RIBES is not damaged very much. We expect this method can be applied to other languages such as German, Korean, ∗This work was done while the second author was a grad- uate student of Okayama Prefectural University. Spearman’s ρ with adequacy NTCIR-7 JE RIBES JE BLEU NTCIR-9 JE RIBES JE BLEU EJ RIBES EJ BLEU NTCIR-10 JE RIBES JE BLEU EJ RIBES EJ BLEU 0.0 0.2 0.4 0.6 0.8 1.0 Figure 1: RIBES has better correlation with ade- quacy than BLEU (system-level correlation) Turkish, Hindi, etc. 1 Introduction For translation among European languages, BLEU (Papineni et al., 2002) has strong cor- relation with human judgments and almost all MT papers use BLEU for evaluation of trans- lation quality. However, BLEU has very weak correlation with human judgments in English-to- Japanese/Japanese-to-English translation, and a new metric RIBES (Isozaki et al., 2010; Hirao et al., 2014) has strong correlation with human judgments. RIBES measures similarity of the word order of a machine translated sentence and that of a human-translated reference sentence. Figure 1 compares RIBES and BLEU in terms of Spearman’s ρ with human judgments of adequacy based on NTCIR-7/9/10 data (Isozaki et al., 2010; Goto et al., 2011; Goto et al., 2013). Japanese and English have completely different word order, and phrase-based SMT systems tend to output bad word orders. RIBES correctly points out their word order problems. In this paper, we propose a method to improve “sentence-level correlation”, which is useful for MT developers to find problems of their MT sys- …

Content truncated to 500 words