Re-evaluating the Role of Bleu in Machine Translation Research

Re-evaluating the Role of BLEU in Machine Translation Research Chris Callison-Burch Miles Osborne Philipp Koehn School on Informatics University of Edinburgh 2 Buccleuch Place Edinburgh, EH8 9LW callison-burch@ed.ac.uk Abstract We argue that the machine translation community is overly reliant on the Bleu machine translation evaluation metric. We show that an improved Bleu score is nei- ther necessary nor sufficient for achieving an actual improvement in translation qual- ity, and give two significant counterex- amples to Bleu’s correlation with human judgments of quality. This offers new po- tential for research which was previously deemed unpromising by an inability to im- prove upon Bleu scores. 1 Introduction Over the past five years progress in machine trans- lation, and to a lesser extent progress in natural language generation tasks such as summarization, has been driven by optimizing against n-gram- based evaluation metrics such as Bleu (Papineni et al., 2002). The statistical machine translation community relies on the Bleu metric for the pur- poses of evaluating incremental system changes and optimizing systems through minimum er- ror rate training (Och, 2003). Conference pa- pers routinely claim improvements in translation quality by reporting improved Bleu scores, while neglecting to show any actual example transla- tions. Workshops commonly compare systems us- ing Bleu scores, often without confirming these rankings through manual evaluation. All these uses of Bleu are predicated on the assumption that it correlates with human judgments of translation quality, which has been shown to hold in many cases (Doddington, 2002; Coughlin, 2003). However, there is a question as to whether min- imizing the error rate with respect to Bleu does in- deed guarantee genuine translation improvements. If Bleu’s correlation with human judgments has been overestimated, then the field needs to ask it- self whether it should continue to be driven by Bleu to the extent that it currently is. In this paper we give a number of counterexamples for Bleu’s correlation with human judgments. We show that under some circumstances an improve- ment in Bleu is not sufficient to reflect a genuine improvement in translation quality, and in other circumstances that it is not necessary to improve Bleu in order to achieve a noticeable improvement in translation quality. We argue that Bleu is insufficient by showing that Bleu admits a huge amount of variation for identically scored hypotheses. Typically there are millions of variations on a hypothesis translation that receive the same Bleu score. Because not all these variations are equally grammatically or se- mantically plausible there are translations which have the same Bleu score but a worse human eval- uation. We further illustrate that in practice a higher Bleu score is not necessarily indicative of better translation quality by giving two substantial examples of Bleu vastly underestimating the trans- lation quality of systems. Finally, we discuss ap- propriate uses for Bleu and suggest that for some research projects it may be preferable to use a fo- cused, manual evaluation instead. 2 BLEU Detailed The rationale behind the development of Bleu (Pa- pineni et al., …

Content truncated to 500 words