A Call for Clarity in Reporting BLEU Scores

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 186–191 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64019 A Call for Clarity in Reporting BLEU Scores Matt Post Amazon Research Berlin, Germany Abstract The field of machine translation faces an under-recognized problem because of incon- sistency in the reporting of scores from its dominant metric. Although people refer to “the” BLEU score, BLEU is in fact a param- eterized metric whose values can vary wildly with changes to these parameters. These pa- rameters are often not reported or are hard to find, and consequently, BLEU scores be- tween papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configu- rations. The main culprit is different tokeniza- tion and normalization schemes applied to the reference. Pointing to the success of the pars- ing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Trans- lation (WMT), which does not allow for user- supplied reference processing, and provide a new tool, SACREBLEU,1 to facilitate this. 1 Introduction Science is the process of formulating hypothe- ses, making predictions, and measuring their out- comes. In machine translation research, the pre- dictions are made by models whose development is the focus of the research, and the measurement, more often than not, is done via BLEU (Papineni et al., 2002). BLEU’s relative language indepen- dence, its ease of computation, and its reason- able correlation with human judgments have led to its adoption as the dominant metric for ma- chine translation research. On the whole, it has been a boon to the community, providing a fast and cheap way for researchers to gauge the perfor- mance of their models. Together with larger-scale controlled manual evaluations, BLEU has shep- 1https://github.com/awslabs/sockeye/ tree/master/contrib/sacrebleu herded the field through a decade and a half of quality improvements (Graham et al., 2014). This is of course not to claim there are no problems with BLEU. Its weaknesses abound, and much has been written about them (cf. Callison- Burch et al. (2006); Reiter (2018)). This paper is not, however, concerned with the shortcomings of BLEU as a proxy for human evaluation of quality; instead, our goal is to bring attention to the rela- tively narrower problem of the reporting of BLEU scores. This problem can be summarized as fol- lows: • BLEU is not a single metric, but requires a number of parameters (§2.1). • Preprocessing schemes have a large effect on scores (§2.2). Importantly, BLEU scores computed against differently-processed ref- erences are not comparable. • Papers vary in the hidden parameters and schemes they use, yet often do not report them (§2.3). Even when they do, it can be hard to discover the details. Together, these issues make it difficult to evaluate and compare BLEU scores across papers, which impedes comparison and replication. I quantify these issues and show that they are serious, with variances bigger …

Content truncated to 500 words