Better Evaluation Metrics Lead to Better Machine Translation

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 375–384, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Better Evaluation Metrics Lead to Better Machine Translation Chang Liu1 and Daniel Dahlmeier2 and Hwee Tou Ng1,2 1Department of Computer Science, National University of Singapore 2NUS Graduate School for Integrative Sciences and Engineering {liuchan1,danielhe,nght}@comp.nus.edu.sg Abstract Many machine translation evaluation met- rics have been proposed after the seminal BLEU metric, and many among them have been found to consistently outperform BLEU, demonstrated by their better correlations with human judgment. It has long been the hope that by tuning machine translation systems against these new generation metrics, ad- vances in automatic machine translation eval- uation can lead directly to advances in auto- matic machine translation. However, to date there has been no unambiguous report that these new metrics can improve a state-of-the- art machine translation system over its BLEU- tuned baseline. In this paper, we demonstrate that tuning Joshua, a hierarchical phrase-based statistical machine translation system, with the TESLA metrics results in significantly better human- judged translation quality than the BLEU- tuned baseline. TESLA-M in particular is simple and performs well in practice on large datasets. We release all our implementation under an open source license. It is our hope that this work will encourage the machine translation community to finally move away from BLEU as the unquestioned default and to consider the new generation metrics when tuning their systems. 1 Introduction The dominant framework of machine translation (MT) today is statistical machine translation (SMT) (Hutchins, 2007). At the core of the system is the decoder, which performs the actual translation. The decoder is parameterized, and estimating the optimal set of parameter values is of paramount importance in getting good translations. In SMT, the parame- ter space is explored by a tuning algorithm, typically MERT (Minimum Error Rate Training) (Och, 2003), though the exact method is not important for our purpose. The tuning algorithm carries out repeated experiments with different decoder parameter val- ues over a development data set, for which reference translations are given. An automatic MT evaluation metric compares the output of the decoder against the reference(s), and guides the tuning algorithm to- wards iteratively better decoder parameters and out- put translations. The quality of the automatic MT evaluation metric therefore has an immediate effect on the whole system. The first automatic MT evaluation metric to show a high correlation with human judgment is BLEU (Papineni et al., 2002). Together with its close vari- ant the NIST metric, they have quickly become the standard way of tuning statistical machine transla- tion systems. While BLEU is an impressively sim- ple and effective metric, recent evaluations have shown that many new generation metrics can out- perform BLEU in terms of correlation with human judgment (Callison-Burch et al., 2009; Callison- Burch et al., 2010). Some of these new metrics in- clude METEOR (Banerjee and Lavie, 2005; Lavie and Agarwal, 2007), TER (Snover et al., 2006), MAXSIM (Chan and …

Content truncated to 500 words