To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 478–494 November 10–11, 2021. ©2021 Association for Computational Linguistics 478 To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation Tom Kocmi Christian Federmann Roman Grundkiewicz Marcin Junczys-Dowmunt Hitokazu Matsushita Arul Menezes Microsoft 1 Microsoft Way Redmond, WA 98052, USA {tomkocmi,chrife,rogrundk,marcinjd,himatsus,arulm}@microsoft.com Abstract Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system’s quality over another. The community choice of automatic metric guides research directions and indus- trial developments by deciding which models are deemed better. Evaluating metrics correla- tions with sets of human judgements has been limited by the size of these sets. In this pa- per, we corroborate how reliable metrics are in contrast to human judgements on – to the best of our knowledge – the largest collection of judgements reported in the literature. Ar- guably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judge- ment as a gold standard, we investigate which metrics have the highest accuracy in predict- ing translation quality rankings for such sys- tem pairs. Furthermore, we evaluate the per- formance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the devel- opment of improved models leading to bad de- ployment decisions. We release the collection of 2.3 M sentence-level human judgements for 4380 systems for further analysis and replica- tion of our work. 1 Introduction Automatic evaluation metrics are commonly used as the main tool for comparing the translation qual- ity of a pair of machine translation (MT) systems (Marie et al., 2021). The decision of which of the two systems is better is often done without the help of human quality evaluation which can be expen- sive and time-consuming. However, as we confirm in this paper, metrics badly approximate human judgement (Mathur et al., 2020b), can be affected by specific phenomena (Zhang and Toral, 2019; Graham et al., 2020; Mathur et al., 2020a; Freitag et al., 2021) or ignore the severity of translation errors (Freitag et al., 2021), and thus may mis- lead system development by incorrect judgements. Therefore, it is important to study the reliability of automatic metrics and follow best practices for the automatic evaluation of systems. Significant research effort has been applied to evaluate automatic metrics in the past decade, in- cluding annual metrics evaluation at the WMT con- ference and other studies (Callison-Burch et al., 2007; Przybocki et al., 2009; Stanojevi´c et al., 2015; Mathur et al., 2020b). Most research has fo- cused on comparing sentence-level (also known as segment-level) correlations between metric scores and human judgements; or system-level (e.g., scor- ing an entire test set) correlations of individual system scores with human judgement. Mathur et al. (2020a) emphasize that this scenario is not identi- cal to the common use of metrics, where instead, researchers and practitioners use automatic scores to compare a pair of …

Content truncated to 500 words