Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 85–91, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems Michael Denkowski and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15232, USA {mdenkows,alavie}@cs.cmu.edu Abstract This paper describes Meteor 1.3, our submis- sion to the 2011 EMNLP Workshop on Sta- tistical Machine Translation automatic evalua- tion metric tasks. New metric features include improved text normalization, higher-precision paraphrase matching, and discrimination be- tween content and function words. We include Ranking and Adequacy versions of the metric shown to have high correlation with human judgments of translation quality as well as a more balanced Tuning version shown to out- perform BLEU in minimum error rate training for a phrase-based Urdu-English system. 1 Introduction The Meteor1 metric (Banerjee and Lavie, 2005; Denkowski and Lavie, 2010b) has been shown to have high correlation with human judgments in eval- uations such as the 2010 ACL Workshop on Statisti- cal Machine Translation and NIST Metrics MATR (Callison-Burch et al., 2010). However, previous versions of the metric are still limited by lack of punctuation handling, noise in paraphrase matching, and lack of discrimination between word types. We introduce new resources for all WMT languages in- cluding text normalizers, filtered paraphrase tables, and function word lists. We show that the addition of these resources to Meteor allows tuning versions of the metric that show higher correlation with human translation rankings and adequacy scores on unseen 1The metric name has previously been stylized as “ME- TEOR” or “METEOR”. As of version 1.3, the official stylization is simply “Meteor”. test data. The evaluation resources are modular, us- able with any other evaluation metric or MT soft- ware. We also conduct a MT system tuning experiment on Urdu-English data to compare the effectiveness of using multiple versions of Meteor in minimum error rate training. While versions tuned to various types of human judgments do not perform as well as the widely used BLEU metric (Papineni et al., 2002), a balanced Tuning version of Meteor consis- tently outperforms BLEU over multiple end-to-end tune-test runs on this data set. The versions of Meteor corresponding to the translation evaluation task submissions, (Ranking and Adequacy), are described in Sections 3 through 5 while the submission to the tunable metrics task, (Tuning), is described in Section 6. 2 New Metric Resources 2.1 Meteor Normalizer Whereas previous versions of Meteor simply strip punctuation characters prior to scoring, version 1.3 includes a new text normalizer intended specifi- cally for translation evaluation. The normalizer first replicates the behavior of the tokenizer distributed with the Moses toolkit (Hoang et al., 2007), includ- ing handling of non-breaking prefixes. After tok- enization, we add several rules for normalization, intended to reduce meaning-equivalent punctuation styles to common forms. The following two rules are particularly helpful: • Remove dashes between hyphenated words. (Example: far-off → far off) 85 • Remove full stops in …

Content truncated to 500 words