Automatic Evaluation Method for Machine Translation Using Noun-Phrase Chunking

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 108–117, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Automatic Evaluation Method for Machine Translation using Noun-Phrase Chunking Hiroshi Echizen-ya Hokkai-Gakuen University S 26-Jo, W 11-chome, Chuo-ku, Sapporo, 064-0926 Japan echi@eli.hokkai-s-u.ac.jp Kenji Araki Hokkaido University N 14-Jo, W 9-Chome, Kita-ku, Sapporo, 060-0814 Japan araki@media.eng.hokudai.ac.jp Abstract As described in this paper, we propose a new automatic evaluation method for machine translation using noun-phrase chunking. Our method correctly deter- mines the matching words between two sentences using corresponding noun phrases. Moreover, our method deter- mines the similarity between two sen- tences in terms of the noun-phrase or- der of appearance. Evaluation experi- ments were conducted to calculate the correlation among human judgments, along with the scores produced us- ing automatic evaluation methods for MT outputs obtained from the 12 ma- 7.chine translation systems in NTCIR- Experimental results show that our method obtained the highest cor- relations among the methods in both sentence-level adequacy and fluency. 1 Introduction High-quality automatic evaluation has be- come increasingly important as various ma- chine translation systems have developed. The scores of some automatic evaluation meth- ods can obtain high correlation with human judgment in document-level automatic evalua- tion(Coughlin, 2007). However, sentence-level automatic evaluation is insufficient. A great gap exists between language processing of au- tomatic evaluation and the processing by hu- mans. Therefore, in recent years, various au- tomatic evaluation methods particularly ad- dressing sentence-level automatic evaluations have been proposed. Methods based on word strings (e.g., BLEU(Papineni et al., 2002), NIST(NIST, 2002), METEOR(Banerjee and Lavie., 2005), ROUGE-L(Lin and Och, 2004), and IMPACT(Echizen-ya and Araki, 2007)) calculate matching scores using only common words between MT outputs and references from bilingual humans. However, these meth- ods cannot determine the correct word corre- spondences sufficiently because they fail to fo- cus solely on phrase correspondences. More- over, various methods using syntactic analyt- ical tools(Pozar and Charniak, 2006; Mutton et al., 2007; Mehay and Brew, 2007) are pro- posed to address the sentence structure. Nev- ertheless, those methods depend strongly on the quality of the syntactic analytical tools. As described herein, for use with MT sys- tems, we propose a new automatic evaluation method using noun-phrase chunking to obtain higher sentence-level correlations. Using noun phrases produced by chunking, our method yields the correct word correspondences and determines the similarity between two sen- tences in terms of the noun phrase order of ap- pearance. Evaluation experiments using MT outputs obtained by 12 machine translation systems in NTCIR-7(Fujii et al., 2008) demon- strate that the scores obtained using our sys- tem yield the highest correlation with the hu- man judgments among the automatic evalua- tion methods in both sentence-level adequacy and fluency. Moreover, the differences be- tween correlation coefficients obtained using our method and other methods are statisti- cally significant at the 5% or lower signifi- cance level for adequacy. Results confirmed that our method using noun-phrase chunking is effective for automatic evaluation for ma- chine translation. 2 …

Content truncated to 500 words