METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support for Five Target Languages

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 339–342, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support for Five Target Languages Michael Denkowski and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15232, USA {mdenkows,alavie}@cs.cmu.edu Abstract This paper describes our submission to the WMT10 Shared Evaluation Task and MetricsMATR10. We present a version of the METEOR-NEXT metric with para- phrase tables for five target languages. We describe the creation of these paraphrase tables and conduct a tuning experiment that demonstrates consistent improvement across all languages over baseline ver- sions of the metric without paraphrase re- sources. 1 Introduction Workshops such as WMT (Callison-Burch et al., 2009) and MetricsMATR (Przybocki et al., 2008) focus on the need for accurate automatic met- rics for evaluating the quality of machine transla- tion (MT) output. While these workshops evalu- ate metric performance on many target languages, most metrics are limited to English due to the rel- ative lack of lexical resources for other languages. This paper describes a language-independent method for adding paraphrase support to the METEOR-NEXT metric for all WMT10 target lan- guages. Taking advantage of the large parallel cor- pora released for the translation tasks often accom- panying evaluation tasks, we automatically con- struct paraphrase tables using the pivot method (Bannard and Callison-Burch, 2005). We use the WMT09 human evaluation data to tune versions of METEOR-NEXT with and without paraphrases and report significantly better performance for ver- sions with paraphrase support. 2 The METEOR-NEXT Metric The METEOR-NEXT metric (Denkowski and Lavie, 2010) evaluates a machine translation hy- pothesis against a reference translation by calcu- lating a similarity score based on an alignment be- tween the two strings. When multiple references are provided, the hypothesis is scored against each and the reference producing the highest score is used. Alignments are formed in two stages: search space construction and alignment selection. For a single hypothesis-reference pair, the space of possible alignments is constructed by identify- ing all possible word and phrase matches between the strings according to the following matchers: Exact: Words are matched if and only if their sur- face forms are identical. Stem: Words are stemmed using a language- appropriate Snowball Stemmer (Porter, 2001) and matched if the stems are identical. Synonym: Words are matched if they are both members of a synonym set according to the Word- Net (Miller and Fellbaum, 2007) database. Paraphrase: Phrases are matched if they are listed as paraphrases in a paraphrase table. The tables used are described in Section 3. Previously, full support has been limited to En- glish, with French, German, and Spanish having exact and stem match support only, and Czech having exact match support only. Although the exact, stem, and synonym match- ers identify word matches while the paraphrase matcher identifies phrase matches, all matches can be generalized to phrase matches with a start po- sition and phrase length in each …

Content truncated to 500 words