Exploring Robustness of Machine Translation Metrics: A Study of Twenty-Two Automatic Metrics in the WMT22 Metric Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 530 - 540 December 7-8, 2022 ©2022 Association for Computational Linguistics Exploring Robustness of Machine Translation Metrics: A Study of Twenty-Eight Automatic Metrics in the WMT22 Metric Task Xiaoyu Chen, Daimeng Wei, Hengchao Shang, Zongyao Li, Zhanglin Wu, Zhengzhe Yu, Ting Zhu, Mengli Zhu, Ning Xie, Lizhi Lei, Shimin Tao, Hao Yang, Ying Qin Huawei Translation Services Center, Beijing, China {chenxiaoyu35,weidaimeng,shanghengchao,lizongyao,wuzhanglin2, yuzhengzhe,zhuting20,zhumengli,nicolas.xie,leilizhi,taoshimin, yanghao30,qinying}@huawei.com Abstract Contextual word embeddings extracted from pre-trained models have become the basis for many downstream NLP tasks, including ma- chine translation automatic evaluations. Met- rics that leverage embeddings claim better cap- ture of synonyms and changes in word orders, and thus better correlation with human rat- ings than surface-form matching metrics (e.g. BLEU). However, few studies have been done to examine robustness of these metrics. This report uses a challenge set to uncover the brit- tleness of reference-based and reference-free metrics. Our challenge set1 aims at examin- ing metrics’ capability to correlate synonyms in different areas and to discern catastrophic errors at both word- and sentence-levels. The results show that although embedding-based metrics perform relatively well on discerning sentence-level negation/affirmation errors, their performances on relating synonyms are poor. In addition, we find that some metrics are sus- ceptible to text styles so their generalizability compromised. 1 Introduction Automatic metrics compare machine-translated results with human-translated references or/and sources, and give scores accordingly. Such met- rics offer a quick and inexpensive approach for re- searchers to evaluate model performances. Among these metrics, BLEU (Papineni et al., 2002) has dominated the area for twenty years since its birth in 2002. However, its limitations are obvious: (1) it weighs each word equally but in fact the entropy of each word varies; (2) it only counts n-grams that are exact in the reference and thus synonyms and elab- orations are wrongly punished (Smith et al., 2016). Consequently, the correlation between BLEU and human evaluation is relatively low, which some- times puzzles researchers. 1We open-source our challenge set at: https://github.com/HwTsc/ Challenge-Set-for-MT-Metrics In recent years, embedding-based approaches have been introduced to design new automatic met- rics. These metrics, e.g. BERTScore (Zhang et al., 2019), COMET (Rei et al., 2020a), and BLEURT (Sellam et al., 2020a), claim better ability to cap- ture synonyms and changes in word order, and thus better performance than BLEU. Apart from ref-based metrics, researches on quality estimation (QE) have been rising, as QE is an cheaper and more convenient approach considering no need of human-translated references. In the WMT metric task, correlation with human annotators is the major indicator to evaluate met- ric performance (Freitag et al., 2021). However, in addition to that, a good metric should meet the following requirements (Banerjee and Lavie, 2005; Koehn, 2009): (1) sensitivity to nuances in quality among systems or outputs of the same system in different stages of its development so it can be used to direct system performance optimization; (2) con- sistency and reliability of scores; (3) usability in a great range …

Content truncated to 500 words