Meteor, M-BLEU and M-TER: Evaluation Metrics for High-Correlation with Human Rankings of Machine Translation Output

Proceedings of the Third Workshop on Statistical Machine Translation, pages 115–118, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Meteor, m-bleu and m-ter: Evaluation Metrics for High-Correlation with Human Rankings of Machine Translation Output Abhaya Agarwal and Alon Lavie Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, 15213, USA {abhayaa,alavie}@cs.cmu.edu Abstract This paper describes our submissions to the machine translation evaluation shared task in ACL WMT-08. Our primary submission is the Meteor metric tuned for optimizing correla- tion with human rankings of translation hy- potheses. We show significant improvement in correlation as compared to the earlier ver- sion of metric which was tuned to optimized correlation with traditional adequacy and flu- ency judgments. We also describe m-bleu and m-ter, enhanced versions of two other widely used metrics bleu and ter respectively, which extend the exact word matching used in these metrics with the flexible matching based on stemming and Wordnet in Meteor . 1 Introduction Automatic Metrics for MT evaluation have been re- ceiving significant attention in recent years. Evalu- ating an MT system using such automatic metrics is much faster, easier and cheaper compared to human evaluations, which require trained bilingual evalua- tors. The most commonly used MT evaluation met- ric in recent years has been IBM’s Bleu metric (Pa- pineni et al., 2002). Bleu is fast and easy to run, and it can be used as a target function in parameter optimization training procedures that are commonly used in state-of-the-art statistical MT systems (Och, 2003). Various researchers have noted, however, var- ious weaknesses in the metric. Most notably, Bleu does not produce very reliable sentence-level scores. Meteor , as well as several other proposed metrics such as GTM (Melamed et al., 2003), TER (Snover et al., 2006) and CDER (Leusch et al., 2006) aim to address some of these weaknesses. Meteor , initially proposed and released in 2004 (Lavie et al., 2004) was explicitly designed to im- prove correlation with human judgments of MT qual- ity at the segment level. Previous publications on Meteor (Lavie et al., 2004; Banerjee and Lavie, 2005; Lavie and Agarwal, 2007) have described the details underlying the metric and have extensively compared its performance with Bleu and several other MT evaluation metrics. In (Lavie and Agar- wal, 2007), we described the process of tuning free parameters within the metric to optimize the corre- lation with human judgments and the extension of the metric for evaluating translations in languages other than English. This paper provides a brief technical description of Meteor and describes our experiments in re-tuning the metric for improving correlation with the human rankings of translation hypotheses corresponding to a single source sentence. Our experiments show sig- nificant improvement in correlation as a result of re- tuning which shows the importance of having a met- ric tunable to different testing conditions. Also, in order to establish the usefulness of the flexible match- ing based on stemming and Wordnet, we extend two other widely used metrics bleu and ter which use …

Content truncated to 500 words