Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 186–199 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 186 Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics Chi-kiu Lo 羅致翹 chikiu.lo@nrc-cnrc.gc.ca Rebecca Knowles rebecca.knowles@nrc-cnrc.gc.ca Cyril Goutte cyril.goutte@nrc-cnrc.gc.ca National Research Council Canada (NRC-CNRC) Abstract While many new automatic metrics for machine translation evaluation have been proposed in recent years, BLEU scores are still used as the primary metric in the vast majority of MT research papers. There are many reasons that researchers may be reluctant to switch to new metrics, from external pressures (reviewers, prior work) to the ease of use of metric toolkits. Another reason is a lack of intuition about the meaning of novel metric scores. In this work, we examine “rules of thumb” about metric score differences and how they do (and do not) correspond to human judgments of statistically significant differences between systems. In particular, we show that common rules of thumb about BLEU score differences do not in fact guarantee that human annotators will find significant differences between systems. We also show ways in which these rules of thumb fail to generalize across translation directions or domains. 1 Introduction Despite mounting evidence over the course of many years (Akiba et al., 2003; Callison-Burch et al., 2006; Chiang et al., 2008; Tan et al., 2015; Mathur et al., 2020a, i.a.) demonstrating that BLEU (Papineni et al., 2002) has fundamental flaws in accurately reflecting translation quality, it has remained the de facto standard automatic MT evaluation metric for both scientific research and practical deployment (Marie et al., 2021). Numerous research efforts (Callison-Burch et al., 2007; Przybocki et al., 2009; Bojar et al., 2017; Freitag et al., 2021b, 2022, i.a.) have focused on the correlations between human judgments of translation quality and automatic metric scores; year after year, these have shown new metrics correlating better with human judgments than BLEU does. There are certainly some other obstacles beyond correlation with human judgment on translation quality that hinder the adoption of newer and better human-correlating automatic MT evaluation metrics in practice. Przybocki et al. (2009) outlined four objectives in the search for new and improved au- tomatic MT evaluation metrics: 1) “high correlation with human assessments of translation quality”; 2) “applicable to multiple target languages”; 3) “ability to differentiate between sys- tems of varying quality” and finally, 4) “intuitive interpretation”, i.e. whether the scores are meaningful and easy to understand on their own, with values and differences that are inter- pretable and clear in practice. As the first three objectives can be addressed by the correlation analysis of MT metrics with human judgment on translation quality but not the last one, we be- lieve that gaining an intuitive understanding of the properties and behavior of the metrics is one 187 of the remaining challenges that MT researchers are facing when they are considering adopting a new metric. …

Content truncated to 500 words