Machine Translation Evaluation with Textual Entailment Features

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 37–41, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Textual Entailment Features for Machine Translation Evaluation Sebastian Pad´o, Michel Galley, Dan Jurafsky, Christopher D. Manning∗ Stanford University {pado,mgalley,jurafsky,manning}@stanford.edu Abstract We present two regression models for the prediction of pairwise preference judgments among MT hy- potheses. Both models are based on feature sets that are motivated by textual entailment and incorporate lexical similarity as well as local syntactic features and specific semantic phenomena. One model pre- dicts absolute scores; the other one direct pairwise judgments. We find that both models are compet- itive with regression models built over the scores of established MT evaluation metrics. Further data analysis clarifies the complementary behavior of the two feature sets. 1 Introduction Automatic metrics to assess the quality of machine trans- lations have been a major enabler in improving the per- formance of MT systems, leading to many varied ap- proaches to develop such metrics. Initially, most metrics judged the quality of MT hypotheses by token sequence match (cf. BLEU (Papineni et al., 2002), NIST (Dod- dington, 2002). These measures rate systems hypothe- ses by measuring the overlap in surface word sequences shared between hypothesis and reference translation. With improvements in the state-of-the-art in machine translation, the effectiveness of purely surface-oriented measures has been questioned (see e.g., Callison-Burch et al. (2006)). In response, metrics have been proposed that attempt to integrate more linguistic information into the matching process to distinguish linguistically li- censed from unwanted variation (Gim´enez and M`arquez, 2008). However, there is little agreement on what types of knowledge are helpful: Some suggestions concen- trate on lexical information, e.g., by the integration of word similarity information as in Meteor (Banerjee and Lavie, 2005) or MaxSim (Chan and Ng, 2008). Other proposals use structural information such as dependency edges (Owczarzak et al., 2007). In this paper, we investigate an MT evaluation metric that is inspired by the similarity between this task and the textual entailment task (Dagan et al., 2005), which ∗This paper is based on work funded by the Defense Ad- vanced Research Projects Agency through IBM. The content does not necessarily reflect the views of the U.S. Government, and no official endorsement should be inferred.. HYP: Virus was infected. REF: No one was infected by the virus. no entailment no entailment HYP: The virus did not infect anybody. REF: No one was infected by the virus. entailment entailment Figure 1: Entailment status between an MT system hy- pothesis and a reference translation for good translations (above) and bad translations (below). suggests that the quality of an MT hypothesis should be predictable by a combination of lexical and structural features that model the matches and mismatches be- tween system output and reference translation. We use supervised regression models to combine these features and analyze feature weights to obtain further insights into the usefulness of different feature types. 2 Textual Entailment for MT Evaluation 2.1 Textual Entailment vs. …

Content truncated to 500 words