Match without a Referee: Evaluating MT Adequacy without Reference Translations

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 171–180, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Match without a Referee: Evaluating MT Adequacy without Reference Translations Yashar Mehdad Matteo Negri Marcello Federico Fondazione Bruno Kessler, FBK-irst Trento , Italy {mehdad|negri|federico}@fbk.eu Abstract We address two challenges for automatic ma- chine translation evaluation: a) avoiding the use of reference translations, and b) focusing on adequacy estimation. From an economic perspective, getting rid of costly hand-crafted reference translations (a) permits to alleviate the main bottleneck in MT evaluation. From a system evaluation perspective, pushing se- mantics into MT (b) is a necessity in order to complement the shallow methods currently used overcoming their limitations. Casting the problem as a cross-lingual textual entail- ment application, we experiment with differ- ent benchmarks and evaluation settings. Our method shows high correlation with human judgements and good results on all datasets without relying on reference translations. 1 Introduction While syntactically informed modelling for statis- tical MT is an active field of research that has re- cently gained major attention from the MT commu- nity, work on integrating semantic models of ade- quacy into MT is still at preliminary stages. This sit- uation holds not only for system development (most current methods disregard semantic information, in favour of statistical models of words distribution), but also for system evaluation. To realize its full po- tential, however, MT is now in the need of semantic- aware techniques, capable of complementing fre- quency counts with meaning representations. In order to integrate semantics more deeply into MT technology, in this paper we focus on the eval- uation dimension. Restricting our investigation to some of the more pressing issues emerging from this area of research, we provide two main contributions. 1. An automatic evaluation method that avoids the use of reference translations. Most current metrics are based on comparisons between auto- matic translations and human references, and reward lexical similarity at the n-gram level (e.g. BLEU (Papineni et al., 2002), NIST (Doddington, 2002), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006)). Due to the variability of natural lan- guages in terms of possible ways to express the same meaning, reliable lexical similarity metrics depend on the availability of multiple hand-crafted (costly) realizations of the same source sentence in the tar- get language. Our approach aims to avoid this bot- tleneck by adapting cross-lingual semantic inference capabilities and judging a translation only given the source sentence. 2. A method for evaluating translation adequacy. Most current solutions do not consistently reward translation adequacy (semantic equivalence between source sentence and target translation). The scarce integration of semantic information in MT, specif- ically at the multilingual level, led to MT systems that are “illiterate” in terms of semantics and mean- ing. Moreover, current metrics are often difficult to interpret. In contrast, our method targets the ade- quacy dimension, producing easily interpretable re- sults (e.g. judgements in a 4-point scale). Our approach builds on recent advances in cross-lingual textual entailment (CLTE) …

Content truncated to 500 words