Statistical Significance Tests for Machine Translation Evaluation

Statistical Significance Tests for Machine Translation Evaluation Philipp Koehn Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology The Stata Center, 32 Vassar Street, Cambridge, MA 02139 koehn@csail.mit.edu Abstract If two translation systems differ differ in perfor- mance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling meth- ods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real. 1 Introduction Recently, the field of machine translation has been changed by the emergence both of effective statisti- cal methods to automatically train machine transla- tion systems from translated text sources (so-called parallel corpora) and of reliable automatic evalua- tion methods. Machine translation systems can now be built and evaluated from black box tools and parallel corpora, with no human involvement at all. The evaluation of machine translation systems has changed dramatically in the last few years. In- stead of reporting human judgment of translation quality, researchers now rely on automatic mea- sures, most notably the BLEU score, which mea- sures n-gram overlap with reference translations. Since it has been shown that the BLEU score cor- relates with human judgment, an improvement in BLEU is taken as evidence for improvement in trans- lation quality. Building the tools for any translation system in- volves many iterations of changes and performance testing. It is important to have a method at hand that gives us assurances that the observed increase in the test score on a test set reflects true improvement in system quality. In other words, we need to be able to gauge, if the increase in score is statistically significant. Since complex metrics such as BLEU do not lend them- selves to an analytical technique for assessing statis- tical significance, we propose bootstrap resampling methods. We also provide empirical evidence that the esti- mated significance levels are accurate by comparing different systems on a large number of test sets of various sizes. In this paper, after providing some background, we will examine some properties of the widely used BLEU metric, discuss experimental design, intro- duce bootstrap resampling methods for statistical significance estimation and report on experimental results that demonstrate the accuracy of the meth- ods. 2 Background 2.1 Statistical Machine Translation Statistical machine translation was introduced by work at IBM [Brown et al., 1990, 1993]. Currently, the most successful such systems employ so-called phrase-based methods that translate input text by translating sequences of words at a time [Och, 2002; Zens et al., 2002; Koehn et al., 2003; Vogel et al., 2003; Tillmann, 2003] Phrase-based machine translation systems make use of a language model trained for the target lan- guage and a translation model trained from a paral- lel corpus. The translation model is typically bro- ken down into several components, e.g., reordering model, phrase translation model, and …

Content truncated to 500 words