Extrinsic Evaluation of Machine Translation Metrics

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13060–13078 July 9-14, 2023 ©2023 Association for Computational Linguistics Extrinsic Evaluation of Machine Translation Metrics Nikita Moghe and Tom Sherborne and Mark Steedman and Alexandra Birch School of Informatics, University of Edinburgh {nikita.moghe, tom.sherborne, a.birch}@ed.ac.uk , steedman@inf.ed.ac.uk Abstract Automatic machine translation (MT) metrics are widely used to distinguish the quality of machine translation systems across large test sets (i.e., system-level evaluation). However, it is unclear if automatic metrics can reliably distinguish good translations from bad at the sentence level (i.e., segment-level evaluation). We investigate how useful MT metrics are at detecting segment-level quality by correlating metrics with the translation utility for down- stream tasks. We evaluate the segment-level performance of widespread MT metrics (chrF, COMET, BERTScore, etc.) on three down- stream cross-lingual tasks (dialogue state track- ing, question answering, and semantic parsing). For each task, we have access to a monolingual task-specific model and a translation model. We calculate the correlation between the met- ric’s ability to predict a good/bad translation with the success/failure on the final task for machine-translated test sentences. Our experi- ments demonstrate that all metrics exhibit neg- ligible correlation with the extrinsic evaluation of downstream outcomes. We also find that the scores provided by neural metrics are not inter- pretable, in large part due to having undefined ranges. We synthesise our analysis into recom- mendations for future MT metrics to produce labels rather than scores for more informative interaction between machine translation and multilingual language understanding. 1 Introduction Although machine translation (MT) is typically seen as a standalone application, in recent years MT models have been more frequently deployed as a component of a complex NLP platform delivering multilingual capabilities such as cross-lingual infor- mation retrieval (Zhang et al., 2022) or automated multilingual customer support (Gerz et al., 2021). When an erroneous translation is generated by the MT systems, it may add new errors in the task pipeline leading to task failure and poor user ex- perience. For example, consider the user’s request in Chinese 剑桥有牙买加菜吗？ (“Is there any good Jamaican food in Cambridge?”) machine- translated into English as “Does Cambridge have a good meal in Jamaica?”. The model will erro- neously consider “Jamaica” as a location, instead of cuisine, and prompt the search engine to look up restaurants in Jamaica 1. To avoid this breakdown, it is crucial to detect an incorrect translation before it causes further errors in the task pipeline. One way to approach this breakdown detection is using segment-level scores provided by MT met- rics. Recent MT metrics have demonstrated high correlation with human judgements at the system level for some language pairs (Ma et al., 2019). These metrics are potentially capable of identifying subtle differences between MT systems that emerge over a relatively large test corpus. These metrics are also evaluated on respective correlation with human judgements at the segment level, however, there is a considerable performance penalty (Ma et al., 2019; Freitag et …

Content truncated to 500 words