Decomposability of Translation Metrics for Improved Evaluation and Efficient Algorithms

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 610–619, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Decomposability of Translation Metrics for Improved Evaluation and Efficient Algorithms David Chiang and Steve DeNeefe Information Sciences Institute University of Southern California 4676 Admiralty Way, Suite 1001 Marina del Rey, CA 90292 USA {chiang,sdeneefe}@isi.edu Yee Seng Chan and Hwee Tou Ng Department of Computer Science National University of Singapore Law Link Singapore 117590 {chanys,nght}@comp.nus.edu.sg Abstract B is the de facto standard for evaluation and development of statistical machine trans- lation systems. We describe three real-world situations involving comparisons between dif- ferent versions of the same systems where one can obtain improvements in B scores that are questionable or even absurd. These situ- ations arise because B lacks the property of decomposability, a property which is also computationally convenient for various appli- cations. We propose a very conservative modi- fication to B and a cross between B and word error rate that address these issues while improving correlation with human judgments. 1 Introduction B (Papineni et al., 2002) was one of the first au- tomatic evaluation metrics for machine translation (MT), and despite being challenged by a number of alternative metrics (Melamed et al., 2003; Baner- jee and Lavie, 2005; Snover et al., 2006; Chan and Ng, 2008), it remains the standard in the statistical MT literature. Callison-Burch et al. (2006) have sub- jected B to a searching criticism, with two real- world case studies of significant failures of corre- lation between B and human adequacy/fluency judgments. Both cases involve comparisons between statistical MT systems and other translation meth- ods (human post-editing and a rule-based MT sys- tem), and they recommend that the use of B be restricted to comparisons between related systems or different versions of the same systems. In B’s de- fense, comparisons between different versions of the same system were exactly what B was designed for. However, we show that even in such situations, difficulties with B can arise. We illustrate three ways that properties of B can be exploited to yield improvements that are questionable or even absurd. All of these scenarios arose in actual prac- tice and involve comparisons between different ver- sions of the same statistical MT systems. They can be traced to the fact that B is not decomposable at the sentence level: that is, it lacks the property that improving a sentence in a test set leads to an increase in overall score, and degrading a sentence leads to a decrease in the overall score. This prop- erty is not only intuitive, but also computationally convenient for various applications such as transla- tion reranking and discriminative training. We pro- pose a minimal modification to B that reduces its nondecomposability, as well as a cross between B and word error rate (WER) that is decompos- able down to the subsentential level (in a sense to be made more precise below). Both metrics correct the observed problems and correlate with human judg- ments …

Content truncated to 500 words