A Study of Translation Edit Rate with Targeted Human Annotation

A Study of Translation Edit Rate with Targeted Human Annotation Matthew Snover and Bonnie Dorr Institute for Advanced Computer Studies University of Maryland College Park, MD 20742 {snover,bonnie}@umiacs.umd.edu Richard Schwartz, Linnea Micciulla, and John Makhoul BBN Technologies 10 Moulton Street Cambridge, MA 02138 {schwartz,lmicciul,makhoul}@bbn.com Abstract We examine a new, intuitive measure for evaluating machine-translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judg- ments. Translation Edit Rate (TER) mea- sures the amount of editing that a hu- man would have to perform to change a system output so it exactly matches a reference translation. We show that the single-reference variant of TER correlates as well with human judgments of MT quality as the four-reference variant of BLEU. We also define a human-targeted TER (or HTER) and show that it yields higher correlations with human judgments than BLEU—even when BLEU is given human-targeted references. Our results in- dicate that HTER correlates with human judgments better than HMETEOR and that the four-reference variants of TER and HTER correlate with human judg- ments as well as—or better than—a sec- ond human judgment does. 1 Introduction Due to the large space of possible correct trans- lations, automatic machine translation (MT) has proved a difficult task to evaluate. Human judg- ments of evaluation are expensive and noisy. Many automatic measures have been proposed to facili- tate fast and cheap evaluation of MT systems, the most widely used of which is BLEU (Papineni et al., 2002), an evaluation metric that matches n- grams from multiple references. A variant of this metric, typically referred to as the “NIST” metric, was proposed by Doddington (Doddington, 2002). Other proposed methods for MT evaluation include METEOR (Banerjee and Lavie, 2005), which uses unigram matches on the words and their stems, and a linear combination of automatic MT evalu- ation methods along with meaning-based features for identifying paraphrases (Russo-Lassner et al., 2005).1 We define a new, more intuitive measure of “goodness” of MT output—specifically, the number of edits needed to fix the output so that it semanti- cally matches a correct translation. In a less expen- sive variant, we attempt to avoid the knowledge in- tensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments. We also seek to achieve higher correlations with human judgments by assigning lower costs to phrasal shifts than those assigned by n-gram-based approaches such as BLEU. Recently the GALE (Olive, 2005) (Global Au- tonomous Language Exploitation) research program introduced a new error measure called Translation Edit Rate (TER)2 that was originally designed to 1One variant of the meaning-based approach incorporates the translation error rate described in this paper. We adopt a simpler evaluation paradigm that requires no meaning-based features, but still achieves correlations that are better than the existing standard, BLEU. 2Within the GALE community, the TER error measure is referred to as Translation Error Rate , derived from the Word Error Rate (WER) metric in the automatic speech recognition community. The name is regrettable for its implication …

Content truncated to 500 words