Self-Supervised Quality Estimation for Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3322–3334 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3322 Self-Supervised Quality Estimation for Machine Translation Yuanhang Zheng1,3,4, Zhixing Tan1,3,4, Meng Zhang6, Mieradilijiang Maimaiti1,3,4, Huanbo Luan1,3,4, Maosong Sun1,3,4,5, Qun Liu6 and Yang Liu∗ 1,2,3,4,5 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Beijing National Research Center for Information Science and Technology 5Beijing Academy of Artificial Intelligence 6Huawei Noah’s Ark Lab Abstract Quality estimation (QE) of machine transla- tion (MT) aims to evaluate the quality of machine-translated sentences without refer- ences and is important in practical applications of MT. Training QE models require massive parallel data with hand-crafted quality anno- tations, which are time-consuming and labor- intensive to obtain. To address the issue of the absence of annotated training data, previ- ous studies attempt to develop unsupervised QE methods. However, very few of them can be applied to both sentence- and word-level QE tasks, and they may suffer from noises in the synthetic data. To reduce the negative im- pact of noises, we propose a self-supervised method for both sentence- and word-level QE, which performs quality estimation by recover- ing the masked target words. Experimental re- sults show that our method outperforms previ- ous unsupervised methods on several QE tasks in different language pairs and domains.1 1 Introduction In recent years, neural approaches (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017) have significantly improved the quality of machine translation (MT). Despite their apparent success, neural machine translation (NMT) systems still inevitably generate erroneous translations in real-world scenarios (Bentivogli et al., 2016; Castilho et al., 2017), especially for low-resource language pairs. Therefore, the eval- uation of translation quality plays an important role in some applications of MT. For example, in computer-assisted translation (CAT) (Barrachina et al., 2009), the evaluation of translation quality can significantly reduce human efforts for post- editing (Specia, 2011). ∗Corresponding author 1Code can be found at https://github.com/ THUNLP-MT/SelfSupervisedQE. Quality estimation (QE) of MT aims to evaluate the quality of the outputs of an MT system without references. Training QE models often requires mas- sive parallel data, which are composed of authentic source sentences and machine-translated target sen- tences with quality annotations produced by man- ual evaluation or human post-editing (Moura et al., 2020; Hu et al., 2020; Ranasinghe et al., 2020). As obtaining such annotated data is time-consuming and labor-intensive in practice, unsupervised QE has received increasing attention (Popovi´c, 2012; Etchegoyhen et al., 2018; Zhang et al., 2020; Zhou et al., 2020; Fomicheva et al., 2020; Tuan et al., 2021). Most of the aforementioned methods use vari- ous features to conduct unsupervised QE (Popovi´c, 2012; Etchegoyhen et al., 2018; Zhang et al., 2020; Zhou et al., 2020; Fomicheva et al., 2020). These methods are simple and effective but limited to sentence-level tasks. Compared with sentence- level QE, word-level QE …

Content truncated to 500 words