Application of Prize based on Sentence Length in Chunk-based Automatic Evaluation of Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 381–386, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Application of Prize based on Sentence Length in Chunk-based Automatic Evaluation of Machine Translation Hiroshi Echizen’ya Hokkai-Gakuen University S26-Jo, W11-Chome, Chuo-ku, Sapporo 064-0926 Japan echi@lst.hokkai-s-u.ac.jp Kenji Araki Hokkaido University N 14-Jo, W 9-Chome, Kita-ku, Sapporo 060-0814 Japan araki@ist.hokudai.ac.jp Eduard Hovy Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 USA hovy@cmu.edu Abstract As described in this paper, we pro- pose a new automatic evaluation met- ric for machine translation. Our met- ric is based on chunking between the reference and candidate translation. Moreover, we apply a prize based on sentence-length to the metric, dissim- ilar from penalties in BLEU or NIST. We designate this metric as Automatic Evaluation of Machine Translation in which the Prize is Applied to a Chunk- based metric (APAC). Through meta- evaluation experiments and compari- son with several metrics, we confirmed that our metric shows stable correla- tion with human judgment. 1 Introduction In the field of machine translation, various automatic evaluation metrics have been pro- posed. Among them, chunk-based metrics such as METEOR(A. Lavie and A. Agarwal, 2007), ROUGE-L(Lin and Och, 2004), and IMPACT(H. Echizen-ya and K. Araki, 2007) are effective. In general, BLEU(K. Papineni et al., 2002), NIST(NIST, 2002), and RIBES(H. Isozaki et al., 2010) use a penalty for calcula- tion of scores because the high score is often given extremely when the candidate transla- tion is short. Therefore, the penalty is effective to obtain high correlation with human judg- ment. On the other hand, almost all chunk- based metrics use the F-measure based on a precision by candidate translation and a re- call by reference. Moreover, they assign a penalty for the difference of chunk order be- tween the candidate translation and the refer- ence, not the penalty for the difference of sen- tence length. Nevertheless, it is also impor- tant for chunk-based metrics to examine the sentence length. In chunk-based metrics, each word’s weight depends on the sentence length. For example, the weight of each word is 0.2 (=1/5) when the number of words in a sen- tence is 5; it is 0.1 (=1/10) when the number of words in a sentence is 10. Therefore, the weight of the non-matched word in the short sentence is large. To resolve this problem, it is effective for short sentences to give a prize based on the sentence length in the chunk-based metrics. Therefore, we propose a new metric using a prize based on the sentence length. We des- ignate this metric as Automatic Evaluation of Machine Translation in which the Prize is Applied to a Chunk-based metric (APAC). In our metric, the weight of a non-matched word becomes small for the short sentence by award- ing of the prize. It is almost identical to that for a long sentence by awarding of the prize. Therefore, our metric does not depend heavily on sentence length because the weight of non- matched words is …

Content truncated to 500 words