Automatic Evaluation Metric for Machine Translation that is Independent of Sentence Length

Proceedings of Recent Advances in Natural Language Processing, pages 230–236, Hissar, Bulgaria, 7-13 September 2013. Automatic Evaluation Metric for Machine Translation that is Independent of Sentence Length Hiroshi Echizen’ya Hokkai-Gakuen University S26-Jo, W11-Chome, Chuo-ku, Sapporo 064-0926 Japan echi@lst.hokkai-s-u.ac.jp Kenji Araki Hokkaido University N 14-Jo, W 9-Chome, Kita-ku, Sapporo 060-0814 Japan araki@media.eng.hokudai.ac.jp Eduard Hovy Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 USA hovy@cmu.edu Abstract We propose new automatic evaluation metric to evaluate machine translation. Different from most similar metrics, our proposed metric does not depend heavily on sentence length. In most metrics based on f-measure compar- isons of reference and candidate trans- lations, the relative weight of each mis- matched word in short sentences is larger than it in long sentences. There- fore, the evaluation score becomes dis- proportionally low in short sentences even when only one non-matching word exists. In our metric, the weight of each mismatched word is kept small even in short sentences. We designate our metric as Automatic Evaluation Metric that is Independent of Sentence Length (AILE). Experimental results indicate that AILE has the highest cor- relation with human judgments among some leading metrics. 1 Introduction Various automatic evaluation metrics for ma- chine translation have been proposed through the metrics task on the Workshop on statis- tical Machine Translation (WMT). One can identify three kinds of automatic evaluation metrics (C. Liu et al., 2010): the heavyweight linguistic approach, which corresponds to RTE (S. Pad´o et al., 2009) and ULC (J. Gim´enez and L. M´arquez, 2007); the lightweight linguis- tic approach, which corresponds to METEOR (A. Lavie and A. Agarwal, 2007) and MaxSim (Y. Seng Chan and H. Tou Ng, 2008) and the non-linguistic approach, which includes BLEU (K. Papineni et al., 2002), TER (M. Snover et al., 2006), RIBES (H. Isozaki et al., 2010) and IMPACT (H. Echizen-ya and K. Araki, 2007)(H. Echizen-ya et al., 2012). In this pa- per, we specifically examine a metric that cor- responds to the lightweight linguistic and non- linguistic approaches because they are useful and are very easily built. Among these metrics, METEOR and IM- PACT are based on the f-measure, which com- bines precision and recall between the refer- ence and candidate texts. The metrics’ simple f-measure (P. Koehn, 2010) obtains precision and recall using Eqs. (1)–(3): precision = matching words length of candidate (1) recall = matching words length of reference (2) Then f-measure is calculated using Eq. (3): f-measure = 2 × precision × recall precision + recall (3) For example, in the reference “doctor cured a patient” and candidate “doctor treated a pa- tient”, the precision and the recall are respec- tively 0.75 (=3 4). Therefore, the f-measure is 0.75 (=2×0.75×0.75 0.75+0.75 ), even though there is only one non-matching word. This is because the denominator is so small, since the sentences 230 are short: the weight of each non-matched word is 0.25 (=1 4) in this example. In general, the relative influence of each non-matching word increases when sentences are short, dis- torting the …

Content truncated to 500 words