METEOR for multiple target languages using DBnary

METEOR for Multiple Target Languages using DBnary. Zied Elloumi zied.elloumi@imag.fr Hervé Blanchon herve.blanchon@imag.fr Gilles Serasset gilles.serasset@imag.fr Laurent Besacier laurent.besacier@imag.fr LIG - GETALP - Univ. Grenoble Alpes Abstract This paper proposes an extension of METEOR, a well-known MT evaluation metric, for multiple target languages using an in-house lexical resource called DBnary (an extraction from Wiktionary provided to the community as a Multilingual Lexical Linked Open Data). Today, the use of the synonymy module of METEOR is only exploited when English is the target language (use of WordNet). A synonymy module using DBnary would allow its use for the 21 languages (covered up to now) as target languages. The code of this new instance of METEOR, adapted to several target languages, is provided to the community. We also show that our DBnary augmented METEOR increases the correlation with human judgements on the WMT 2013 and 2014 metrics dataset for English-to-(French, Russian, German, Spanish) language pairs. 1. Introduction Machine translation (MT) is the process of automatically translating a text in a source language into a corresponding text in a target language. In order to evaluate and compare the quality of several MT systems, we need to rate the translation hypothesis produced by each MT system either with the help of human experts (subjective evaluation) or compare it to pre- existing human translations (using automatic evaluation metrics, objective evaluation). In practice, subjective evaluation considers various aspects to grade the translation quality, such as adequacy, fluency, and intelligibility. However, subjective evaluation conducted a posteriori often costs too much (in term of human resources) and, thus, objective evaluation metrics (fast and cheap as long as references are available) are often preferred nowadays. One drawback with automatic evaluation metrics is that they compare the MT hypothesis with few (and sometimes only one) reference translations. This is definitely not enough to capture lexical variation in translation. For this reason, metrics which exploit synonymy or stem similarities, such as METEOR (Banerjee and Lavie, 2005), exhibit better correlation with human judgement. METEOR maps words with the same stem or the same synset using lexico-semantic resources. However, so far, the full potential of METEOR is only exploited when English is the target language (use of WordNet). Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 80 Contribution This paper proposes an extension of METEOR for multiple target languages using a lexical resource called DBnary (Sérasset, 2015). DBnary is an extraction in RDF of the lexical data of multiple editions of Wiktionary. It has several millions of triples describing lexical entries of the extracted languages, and more than 4.6 million translations from 21 languages to more than 1500 target languages. The modified code allowing to call METEOR for new target languages (French, Russian, German, Spanish) is made available to the research community. More target languages (today 21 in total) could be plugged very quickly by interested users using the same lexical resource (DBnary notably includes Bulgarian, Dutch, English, Finnish, French, German, (Modern) Greek, …

Content truncated to 500 words