Robust Machine Translation Evaluation with Entailment Features

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 297–305, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Robust Machine Translation Evaluation with Entailment Features∗ Sebastian Pad´o Stuttgart University pado@ims.uni-stuttgart.de Michel Galley, Dan Jurafsky, Chris Manning Stanford University {mgalley,jurafsky,manning}@stanford.edu Abstract Existing evaluation metrics for machine translation lack crucial robustness: their correlations with hu- man quality judgments vary considerably across lan- guages and genres. We believe that the main reason is their inability to properly capture meaning: A good translation candidate means the same thing as the reference translation, regardless of formulation. We propose a metric that evaluates MT output based on a rich set of features motivated by textual entailment, such as lexical-semantic (in-)compatibility and ar- gument structure overlap. We compare this metric against a combination metric of four state-of-the- art scores (BLEU, NIST, TER, and METEOR) in two different settings. The combination metric out- performs the individual scores, but is bested by the entailment-based metric. Combining the entailment and traditional features yields further improvements. 1 Introduction Constant evaluation is vital to the progress of ma- chine translation (MT). Since human evaluation is costly and difficult to do reliably, a major focus of research has been on automatic measures of MT quality, pioneered by BLEU (Papineni et al., 2002) and NIST (Doddington, 2002). BLEU and NIST measure MT quality by using the strong correla- tion between human judgments and the degree of n-gram overlap between a system hypothesis trans- lation and one or more reference translations. The resulting scores are cheap and objective. However, studies such as Callison-Burch et al. (2006) have identified a number of problems with BLEU and related n-gram-based scores: (1) BLEU- like metrics are unreliable at the level of individual sentences due to data sparsity; (2) BLEU metrics can be “gamed” by permuting word order; (3) for some corpora and languages, the correlation to hu- man ratings is very low even at the system level; (4) scores are biased towards statistical MT; (5) the quality gap between MT and human translations is not reflected in equally large BLEU differences. ∗This paper is based on work funded by the Defense Ad- vanced Research Projects Agency through IBM. The content does not necessarily reflect the views of the U.S. Government, and no official endorsement should be inferred. This is problematic, but not surprising: The met- rics treat any divergence from the reference as a negative, while (computational) linguistics has long dealt with linguistic variation that preserves the meaning, usually called paraphrase, such as: (1) HYP: However, this was declared terrorism by observers and witnesses. REF: Nevertheless, commentators as well as eyewitnesses are terming it terrorism. A number of metrics have been designed to account for paraphrase, either by making the matching more intelligent (TER, Snover et al. (2006)), or by using linguistic evidence, mostly lexical similarity (ME- TEOR, Banerjee and Lavie (2005); MaxSim, Chan and Ng (2008)), or syntactic overlap (Owczarzak et al. (2008); Liu and Gildea (2005)). Unfortunately, …

Content truncated to 500 words