Randomized Significance Tests in Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266–274, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Randomized Significance Tests in Machine Translation Yvette Graham Nitika Mathur Timothy Baldwin Department of Computing and Information Systems The University of Melbourne ygraham@unimelb.edu.au, nmathur@student.unimelb.edu.au, tb@ldwin.net Abstract Randomized methods of significance test- ing enable estimation of the probability that an increase in score has occurred sim- ply by chance. In this paper, we examine the accuracy of three randomized meth- ods of significance testing in the context of machine translation: paired bootstrap resampling, bootstrap resampling and ap- proximate randomization. We carry out a large-scale human evaluation of shared task systems for two language pairs to provide a gold standard for tests. Re- sults show very little difference in accu- racy across the three methods of signif- icance testing. Notably, accuracy of all test/metric combinations for evaluation of English-to-Spanish are so low that there is not enough evidence to conclude they are any better than a random coin toss. 1 Introduction Automatic metrics, such as BLEU (Papineni et al., 2002), are widely used in machine translation (MT) as a substitute for human evaluation. Such metrics commonly take the form of an automatic comparison of MT output text with one or more human reference translations. Small differences in automatic metric scores can be difficult to inter- pret, however, and statistical significance testing provides a way of estimating the likelihood that a score difference has occurred simply by chance. For several metrics, such as BLEU, standard sig- nificance tests cannot be applied due to scores not comprising the mean of individual sentence scores, justifying the use of randomized methods. Bootstrap resampling was one of the early ran- domized methods proposed for statistical signifi- cance testing of MT (Germann, 2003; Och, 2003; Kumar and Byrne, 2004; Koehn, 2004), to assess for a pair of systems how likely a difference in BLEU scores occurred by chance. Empirical tests detailed in Koehn (2004) show that even for test sets as small as 300 translations, BLEU confidence intervals can be computed as accurately as if they had been computed on a test set 100 times as large. Approximate randomization was subsequently proposed as an alternate to bootstrap resam- pling (Riezler and Maxwell, 2005). Theoretically speaking, approximate randomization has an ad- vantage over bootstrap resampling, in that it does not make the assumption that samples are repre- sentative of the populations from which they are drawn. Both methods require some adaptation in order to be used for the purpose of MT evalua- tion, such as combination with an automatic met- ric, and therefore it cannot be taken for granted that approximate randomization will be more ac- curate in practice. Within MT, approximate ran- domization for the purpose of statistical testing is also less common. Riezler and Maxwell (2005) provide a compar- ison of approximate randomization with bootstrap resampling (distinct from paired bootstrap resam- pling), and conclude that since approximate ran- domization produces higher p-values for a set …

Content truncated to 500 words