Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Meteor Universal: Language Specific Translation Evaluation for Any Target Language Michael Denkowski Alon Lavie Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 USA {mdenkows,alavie}@cs.cmu.edu Abstract This paper describes Meteor Universal, re- leased for the 2014 ACL Workshop on Statistical Machine Translation. Meteor Universal brings language specific evalu- ation to previously unsupported target lan- guages by (1) automatically extracting lin- guistic resources (paraphrase tables and function word lists) from the bitext used to train MT systems and (2) using a univer- sal parameter set learned from pooling hu- man judgments of translation quality from several language directions. Meteor Uni- versal is shown to significantly outperform baseline BLEU on two new languages, Russian (WMT13) and Hindi (WMT14). 1 Introduction Recent WMT evaluations have seen a variety of metrics employ language specific resources to replicate human translation rankings far better than simple baselines (Callison-Burch et al., 2011; Callison-Burch et al., 2012; Mach´aˇcek and Bojar, 2013; Snover et al., 2009; Denkowski and Lavie, 2011; Dahlmeier et al., 2011; Chen et al., 2012; Wang and Manning, 2012, inter alia). While the wealth of linguistic resources for the WMT lan- guages allows the development of sophisticated metrics, most of the world’s 7,000+ languages lack the prerequisites for building advanced metrics. Researchers working on low resource languages are usually limited to baseline BLEU (Papineni et al., 2002) for evaluating translation quality. Meteor Universal brings language specific eval- uation to any target language by combining lin- guistic resources automatically learned from MT system training data with a universal metric pa- rameter set that generalizes across languages. Given only the bitext used to build a standard phrase-based translation system, Meteor Universal learns a paraphrase table and function word list, two of the most consistently beneficial language specific resources employed in versions of Me- teor. Whereas previous versions of Meteor require human ranking judgments in the target language to learn parameters, Meteor Universal uses a sin- gle parameter set learned from pooling judgments from several languages. This universal parameter set captures general preferences shown by human evaluators across languages. We show this ap- proach to significantly outperform baseline BLEU for two new languages, Russian and Hindi. The following sections review Meteor’s scoring func- tion (§2), describe the automatic extraction of lan- guage specific resources (§3), discuss training of the universal parameter set (§4), report experimen- tal results (§5), describe released software (§6), and conclude (§7). 2 Meteor Scoring Meteor evaluates translation hypotheses by align- ing them to reference translations and calculating sentence-level similarity scores. For a hypothesis- reference pair, the space of possible alignments is constructed by exhaustively identifying all possi- ble matches between the sentences according to the following matchers: Exact: Match words if their surface forms are identical. Stem: Stem words using a language appropriate Snowball Stemmer (Porter, 2001) and match if the stems are identical. Synonym: Match words if they share member- ship in …

Content truncated to 500 words