Quality Estimation Using Dual Encoders with Transfer Learning

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 920–927 November 10–11, 2021. ©2021 Association for Computational Linguistics 920 Quality Estimation Using Dual Encoders with Transfer Learning Dam Heo∗ WonKee Lee∗ Baikjin Jung∗ Jong-Hyeok Lee∗,† ∗Departmet of Computer Science and Engineering, †Graduate School of Artificial Intelligence, Pohang University of Science and Technology (POSTECH), Republic of Korea {dammy, wklee, bjjung, jhlee}@postech.ac.kr Abstract This paper describes POSTECH’s quality es- timation systems submitted to Task 2 of the WMT 2021 quality estimation shared task: Word and Sentence-Level Post-editing Effort. We aim to improve the stability of recently pro- posed quality estimation models, which usu- ally have a single encoder based on the self- attention mechanism to simultaneously pro- cess both of the two input data: a source se- quence and its machine translation; consid- ering that such models are not propped up by pre-trained language models’ monolingual word representations, which are generally ac- cepted as reliable representations for various natural language processing tasks. Therefore, our model first uses two pre-trained monolin- gual encoders and then exchanges their out- put information through two additional cross attention networks. According to the official leaderboard, our systems outperform the base- line systems in terms of the Matthews correla- tion coefficient for machine translations’ word- level quality estimation and in terms of the Pearson’s correlation coefficient for sentence- level quality estimation by 0.4126 and 0.5497 respectively. 1 Introduction Quality estimation (QE) is the task of estimating the quality of given machine translations without regard to their reference translations (Blatz et al., 2004; Specia et al., 2009). As reference translations are generally unavailable in real life, QE should help to treat output texts of machine translation (MT) systems. QE can be categorized into sev- eral subtasks, and this round of the WMT QE task has three subtasks, yet we focus on Task 2: Word and Sentence-Level Post-editing Effort. In Task 2, while sentence-level QE aims to predict the Human- Targeted Translation Edit Rate (HTER, Snover et al. 2006), which measures the edit distance between an MT output (mt) and its human post-edited text (pe), word-level QE aims to predict OK–BAD tags for three sequences of tokens: the sequence of words in a source text (src) depending on whether they are correctly translated referring to mt; the sequence of words in mt depending on their correctness; and <GAP> tokens, which each represent the gap be- tween two adjacent words, depending on the exis- tence of any missing words (Specia et al., 2020). As other recent QE models do, our method also applies transfer learning, considering that pre- trained language models (LM) have been success- fully applied to various natural language processing (NLP) tasks including QE; many previous studies (Fomicheva et al., 2020; Hu et al., 2020; Wu et al., 2020; Lee, 2020; Moura et al., 2020; Nakamachi et al., 2020; Rubino, 2020) that apply pre-trained LMs to QE have adopted multilingual or cross- lingual LMs such as multilingual-BERT (Pires et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R …

Content truncated to 500 words