Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4984 Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics Nitika Mathur Timothy Baldwin Trevor Cohn School of Computing and Information Systems The University of Melbourne Victoria 3010, Australia nmathur@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au Abstract Automatic metrics are fundamental for the de- velopment and evaluation of machine transla- tion systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straight- forward problem. We show that current meth- ods for judging metrics are highly sensitive to the translations used for assessment, par- ticularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pair- wise system ranking, developing a method for thresholding performance improvement un- der an automatic metric against human judge- ments, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings sug- gest improvements to the protocols for metric evaluation and system performance evaluation in machine translation. 1 Introduction Automatic metrics are an indispensable part of machine translation (MT) evaluation, serving as a proxy to human evaluation which is consider- ably more expensive and time-consuming. They provide immediate feedback during MT system development and serve as the primary metric to report the quality of MT systems. Accordingly, the reliability of metrics is critical to progress in MT research. A particularly worrying finding was made in the most recent Conference on Machine Translation (WMT), as part of their annual competition find- ings to benchmark progress in translation and trans- lation evaluation. WMT has established a method based on Pearson’s correlation coefficient for mea- suring how well automatic metrics match with hu- man judgements of translation quality, which is used to rank metrics and to justify their widespread use in lieu of human evaluation. Their findings (Ma et al., 2019) showed that if the correlation is computed for metrics using a large cohort of transla- tion systems, typically very high correlations were found between leading metrics and humans (as high as r = 0.9). However, if considering only the few best systems, the correlation reduced markedly. This is in contrast to findings at sentence-level eval- uation, where metrics are better at distinguishing between high-quality translations compared to low- quality translations (Fomicheva and Specia, 2019). When considering only the four best systems, the automatic metrics were shown to exhibit negative correlations in some instances. It would appear that metrics can only be relied upon for making coarse distinctions between poor and good translation out- puts, but not for assessing similar quality outputs, i.e., the most common application faced when as- sessing incremental empirical improvements. Overall these findings raise important questions as to the reliability of the accepted best-practises for ranking metrics, and more fundamentally, cast doubt over …

Content truncated to 500 words