BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5428–5443 July 9-14, 2023 ©2023 Association for Computational Linguistics BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training Yiming Yan1∗, Tao Wang2, Chengqi Zhao2, Shujian Huang1†, Jiajun Chen1, Mingxuan Wang2 1 National Key Laboratory for Novel Software Technology, Nanjing University, China 2 ByteDance AI Lab, China yanym@smail.nju.edu.cn, {huangsj, chenjj}@nju.edu.cn {wangtao.960826, zhaochengqi.d, wangmingxuan.89}@bytedance.com Abstract Automatic metrics play a crucial role in ma- chine translation. Despite the widespread use of n-gram-based metrics, there has been a re- cent surge in the development of pre-trained model-based metrics that focus on measuring sentence semantics. However, these neural met- rics, while achieving higher correlations with human evaluations, are often considered to be black boxes with potential biases that are diffi- cult to detect. In this study, we systematically analyze and compare various mainstream and cutting-edge automatic metrics from the per- spective of their guidance for training machine translation systems. Through Minimum Risk Training (MRT), we find that certain metrics ex- hibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm. By incorporating token-level constraints, we en- hance the robustness of evaluation metrics, which in turn leads to an improvement in the performance of machine translation systems. Codes are available at https://github.com/ powerpuffpomelo/fairseq_mrt. 1 Introduction Automatic metrics are crucial for the training of machine translation models, as they can measure translation quality at low cost. Currently, the most widely used translation evaluation metric is still the n-gram-based BLEU (Papineni et al., 2002; Marie et al., 2021). However, it is acknowledged that BLEU, which relies on the surface-level vo- cabulary matching, exhibits significant limitations (Smith et al., 2016; Reiter, 2018; Mathur et al., 2020; Kocmi et al., 2021). For instance, BLEU fails to differentiate between errors of varying severity and assigns equal weight to each word. *Work was done during internship at ByteDance AI Lab. †Corresponding author. Figure 1: An example of a universal adversarial trans- lation of BLEURT. hypo means the translation sen- tence and ref means the reference sentence. BLEURT needs to compare hypo and ref to judge the quality of hypo. This figure shows that the universal translation can achieve high BLEURT scores when calculated with each ref, even if hypo and ref are completely unre- lated. In recent years, the advent of pre-trained mod- els (Devlin et al., 2018; Liu et al., 2019; Conneau et al., 2019; Yang et al., 2019; Lan et al., 2019) has led to significant advancements in the development of metrics such as BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020), which employ pre- trained language models (PLM) to assess the se- mantic meaning of sentences. These approaches have been shown to outperform metrics that rely on superficial word matching and have a more con- sistent correlation …

Content truncated to 500 words