METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, June 2007. c⃝2007 Association for Computational Linguistics Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments Alon Lavie and Abhaya Agarwal Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, 15213, USA {alavie,abhayaa}@cs.cmu.edu Abstract Meteor is an automatic metric for Ma- chine Translation evaluation which has been demonstrated to have high levels of corre- lation with human judgments of translation quality, significantly outperforming the more commonly used Bleu metric. It is one of several automatic metrics used in this year’s shared task within the ACL WMT-07 work- shop. This paper recaps the technical de- tails underlying the metric and describes re- cent improvements in the metric. The latest release includes improved metric parameters and extends the metric to support evalua- tion of MT output in Spanish, French and German, in addition to English. 1 Introduction Automatic Metrics for MT evaluation have been re- ceiving significant attention in recent years. Evalu- ating an MT system using such automatic metrics is much faster, easier and cheaper compared to human evaluations, which require trained bilingual evalua- tors. Automatic metrics are useful for comparing the performance of different systems on a common translation task, and can be applied on a frequent and ongoing basis during MT system development. The most commonly used MT evaluation metric in recent years has been IBM’s Bleu metric (Papineni et al., 2002). Bleu is fast and easy to run, and it can be used as a target function in parameter op- timization training procedures that are commonly used in state-of-the-art statistical MT systems (Och, 2003). Various researchers have noted, however, var- ious weaknesses in the metric. Most notably, Bleu does not produce very reliable sentence-level scores. Meteor , as well as several other proposed metrics such as GTM (Melamed et al., 2003), TER (Snover et al., 2006) and CDER (Leusch et al., 2006) aim to address some of these weaknesses. Meteor , initially proposed and released in 2004 (Lavie et al., 2004) was explicitly designed to im- prove correlation with human judgments of MT qual- ity at the segment level. Previous publications on Meteor (Lavie et al., 2004; Banerjee and Lavie, 2005) have described the details underlying the met- ric and have extensively compared its performance with Bleu and several other MT evaluation met- rics. This paper recaps the technical details underly- ing Meteor and describes recent improvements in the metric. The latest release extends Meteor to support evaluation of MT output in Spanish, French and German, in addition to English. Furthermore, several parameters within the metric have been opti- mized on language-specific training data. We present experimental results that demonstrate the improve- ments in correlations with human judgments that re- sult from these parameter tunings. 2 The Meteor Metric Meteor evaluates a translation by computing a score based on explicit word-to-word matches be- tween the translation and a given reference trans- lation. If more than one reference translation is available, the …

Content truncated to 500 words