Alignment-based sense selection in METEOR and the RATATOUILLE recipe

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 385–391, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Alignment-based sense selection in METEOR and the RATATOUILLE recipe Benjamin Marie LIMSI-CNRS, Orsay, France Lingua et Machina, Le Chesnay, France benjamin.marie@limsi.fr Marianna Apidianaki LIMSI-CNRS, Orsay, France marianna@limsi.fr Abstract This paper describes Meteor-WSD and RATATOUILLE, the LIMSI submissions to the WMT15 metrics shared task. Meteor- WSD extends synonym mapping to lan- guages other than English based on align- ments and gives credit to semantically adequate translations in context. We show that context-sensitive synonym se- lection increases the correlation of the Meteor metric with human judgments of translation quality on the WMT14 data. RATATOUILLE combines Meteor- WSD with nine other metrics for eval- uation and outperforms the best metric (BEER) involved in its computation. 1 Introduction The Meteor metric evaluates translation hypothe- ses by aligning them to reference translations and calculating sentence-level similarity scores (Banerjee and Lavie, 2005; Denkowski and Lavie, 2010). The space of possible alignments for a hypothesis-reference pair is constructed by identi- fying all possible matches between the sentences according to different matchers mapping words with identical surface forms or having the same stem, WordNet synonyms and paraphrases. These modules add flexibility to the metric and im- prove its correlation with human judgments of translation quality but they fail to account for important semantics-related aspects. For exam- ple, Meteor and Meteor-NEXT treat all the vari- ants available for a particular text fragment in WordNet (Fellbaum, 1998) or a pivot paraphrase database (Bannard and Callison-Burch, 2005) as semantically equivalent. Consequently, erroneous matches can be made by mapping synonyms found in different WordNet synsets and describing dif- ferent senses. Similarly, pivot paraphrase sets merge sense boundaries in cases of polysemous words (Apidianaki et al., 2014), which means that paraphrases of different senses are considered as equivalent and can be mapped during evaluation. To avoid erroneous matches between text seg- ments, it is thus important to restrict the available word and phrase variants to the ones that are cor- rect in a specific context. Context-based synonym selection is the main idea behind the Meteor-WSD metric submitted to the WMT15 Metrics Shared Task. The mecha- nism used for sense selection is described in detail in the next section where we also present the re- sults obtained by the Meteor-WSD metric on the WMT14 evaluation dataset. Section 3 presents the RATATOUILLE metric which integrates Meteor- WSD together with nine other evaluation metrics. We report results in all language pairs and direc- tions of the WMT14 dataset, except for hi-en. 2 Meteor-WSD 2.1 Context-dependent sense selection A first attempt to integrate context-based sense se- lection in Meteor is described in Apidianaki and Marie (2015). Word sense disambiguation (WSD) was performed using the Babelfy tool (Moro et al., 2014) which relies on the multilingual resource BabelNet (Navigli and Ponzetto, 2012). BabelNet is a wide coverage semantic network where senses are described by synsets (synonym and paraphrase sets) containing lexicographic and encyclopedic knowledge extracted from various …

Content truncated to 500 words