Improving Translation Quality Estimation with Bias Mitigation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2175–2190 July 9-14, 2023 ©2023 Association for Computational Linguistics Improving Translation Quality Estimation with Bias Mitigation Hui Huang1†, Shuangzhi Wu2, Kehai Chen3, Hui Di4, Muyun Yang1‡, Tiejun Zhao1 1Faculty of Computing, Harbin Institute of Technology, Harbin, China 2ByteDance AI Lab, Beijing, China 3School of Computer Science and Technology, Harbin Institute of Technolgy, Shenzhen, China 4Research&Development Center, Toshiba (China) Co., Ltd, Beijing, China 22b903058@stu.hit.edu.cn, wufurui@bytedance.com, dihui@toshiba.com.cn, {chenkehai, yangmuyun, tjzhao}@hit.edu.cn; Abstract State-of-the-art translation Quality Estimation (QE) models are proven to be biased. More specifically, they over-rely on monolingual fea- tures while ignoring the bilingual semantic alignment. In this work, we propose a novel method to mitigate the bias of the QE model and improve estimation performance. Our method is based on the contrastive learning be- tween clean and noisy sentence pairs. We first introduce noise to the target side of the paral- lel sentence pair, forming the negative samples. With the original parallel pairs as the positive sample, the QE model is contrastively trained to distinguish the positive samples from the negative ones. This objective is jointly trained with the regression-style quality estimation, so as to prevent the QE model from overfitting to monolingual features. Experiments on WMT QE evaluation datasets demonstrate that our method improves the estimation performance by a large margin while mitigating the bias1. 1 Introduction Quality Estimation (QE) aims to predict the qual- ity of machine translation automatically in the ab- sence of reference translations. State-of-the-art QE model mostly falls into Pre-Trained Model (PTM)- based paradigm. In the latest QE evaluation tasks (Zerva et al., 2022), nearly all top-performing sys- tems adopt Multilingual PTMs as backbone. Good as the PTM based QE performance is, re- cent researches (Sun et al., 2020; Behnke et al., 2022) reveal that state-of-the-art QE models are biased. To be specific, the models largely rely on spurious monolingual features, such as the flu- ency of the target sequence, or the complexity of the source sequence, without really capturing the †Contribution during internship at ByteDance Inc. ‡Corresponding Authors. 1Codes are available at https://github.com/ HuihuiChyan/AwesomeQE-contrast Figure 1: An example of the bias in QE. Notice the first sentence pair is unrelated. bilingual semantic alignment. Such monolingual features do not have a casual impact on the trans- lation quality, and bias the QE results to a large extent. For example, as shown in Figure 1, a fluent and uncomplicated translation might be assigned with a high quality score even it does not resemble the actual semantics of the source sentence, while an adequate translation with complicated structure might be assigned as bad translation. Sun et al. (2020) recommends to counter with the bias by using a metric that represents adequacy well as labels. However, in their such annotated dataset, the bias is still striking, as revealed by Behnke et al. (2022). As an alternative, Behnke et al. (2022) ex- plores several multitask architectures, to support the QE task and discourage …

Content truncated to 500 words