TINE: A Metric to Assess MT Adequacy

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 116–122, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics TINE: A Metric to Assess MT Adequacy Miguel Rios, Wilker Aziz and Lucia Specia Research Group in Computational Linguistics University of Wolverhampton Stafford Street, Wolverhampton, WV1 1SB, UK {m.rios, w.aziz, l.specia}@wlv.ac.uk Abstract We describe TINE, a new automatic evalua- tion metric for Machine Translation that aims at assessing segment-level adequacy. Lexical similarity and shallow-semantics are used as indicators of adequacy between machine and reference translations. The metric is based on the combination of a lexical matching com- ponent and an adequacy component. Lexi- cal matching is performed comparing bags- of-words without any linguistic annotation. The adequacy component consists in: i) us- ing ontologies to align predicates (verbs), ii) using semantic roles to align predicate argu- ments (core arguments and modifiers), and iii) matching predicate arguments using dis- tributional semantics. TINE’s performance is comparable to that of previous metrics at segment level for several language pairs, with average Kendall’s tau correlation from 0.26 to 0.29. We show that the addition of the shallow-semantic component improves the performance of simple lexical matching strate- gies and metrics such as BLEU. 1 Introduction The automatic evaluation of Machine Translation (MT) is a long-standing problem. A number of met- rics have been proposed in the last two decades, mostly measuring some form of matching between the MT output (hypothesis) and one or more human (reference) translations. However, most of these metrics focus on fluency aspects, as opposed to ad- equacy. Therefore, measuring whether the meaning of the hypothesis and reference translation are the same or similar is still an understudied problem. The most commonly used metrics, BLEU (Pap- ineni et al., 2002) and alike, perform simple exact matching of n-grams between hypothesis and refer- ence translations. Such a simple matching proce- dure has well known limitations, including that the matching of non-content words counts as much as the matching of content words, that variations of words with the same meaning are disregarded, and that a perfect matching can happen even if the order of sequences of n-grams in the hypothesis and ref- erence translation are very different, changing com- pletely the meaning of the translation. A number of other metrics have been proposed to address these limitations, for example, by allow- ing for the matching of synonyms or paraphrases of content words, such as in METEOR (Denkowski and Lavie, 2010). Other attempts have been made to capture whether the reference translation and hy- pothesis translations share the same meaning us- ing shallow semantics, i.e., Semantic Role Labeling (Gim´enez and M´arquez, 2007). However, these are limited to the exact matching of semantic roles and their fillers. We propose TINE, a new metric that comple- ments lexical matching with a shallow semantic component to better address adequacy. The main contribution of such a metric is to provide a more flexible way of measuring the overlap between shal- low semantic representations that considers both the semantic …

Content truncated to 500 words