Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 250–253, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level Michael Denkowski and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15232, USA {mdenkows,alavie}@cs.cmu.edu Abstract This paper presents METEOR-NEXT, an ex- tended version of the METEOR metric de- signed to have high correlation with post- editing measures of machine translation qual- ity. We describe changes made to the met- ric’s sentence aligner and scoring scheme as well as a method for tuning the metric’s pa- rameters to optimize correlation with human- targeted Translation Edit Rate (HTER). We then show that METEOR-NEXT improves cor- relation with HTER over baseline metrics, in- cluding earlier versions of METEOR, and ap- proaches the correlation level of a state-of-the- art metric, TER-plus (TERp). 1 Introduction Recent focus on the need for accurate automatic metrics for evaluating the quality of machine trans- lation output has spurred much development in the field of MT. Workshops such as WMT09 (Callison- Burch et al., 2009) and the MetricsMATR08 chal- lenge (Przybocki et al., 2008) encourage the devel- opment of new MT metrics and reliable human judg- ment tasks. This paper describes our work extending the ME- TEOR metric to improve correlation with human- targeted Translation Edit Rate (HTER) (Snover et al., 2006), a semi-automatic post-editing based met- ric which measures the distance between MT out- put and a targeted reference. We identify several limitations of the original METEOR metric and de- scribe our modifications to improve performance on this task. Our extended metric, METEOR-NEXT, is then tuned to maximize segment-level correlation with HTER scores and tested against several base- line metrics. We show that METEOR-NEXT outper- forms earlier versions of METEOR when tuned to the same HTER data and approaches the performance of a state-of-the-art TER-based metric, TER-plus. 2 The METEOR-NEXT Metric 2.1 Traditional METEOR Scoring Given a machine translation hypothesis and a refer- ence translation, the traditional METEOR metric cal- culates a lexical similarity score based on a word- to-word alignment between the two strings (Baner- jee and Lavie, 2005). When multiple references are available, the hypothesis is scored against each and the reference producing the highest score is used. Alignments are built incrementally in a series of stages using the following METEOR matchers: Exact: Words are matched if and only if their sur- face forms are identical. Stem: Words are stemmed using a language- appropriate Snowball Stemmer (Porter, 2001) and matched if the stems are identical. Synonym: Words are matched if they are both members of a synonym set according to the Word- Net (Miller and Fellbaum, 2007) database. matcher is limited to translations into English.This At each stage, one of the above matchers iden- tifies all possible word matches between the two translations using words not aligned in previous stages. An alignment is then identified as the largest subset of these matches …

Content truncated to 500 words