Revisiting Round-trip Translation for Quality Estimation

Revisiting Round-Trip Translation for Quality Estimation Jihyung Moon Naver Papago Hyunchang Cho Naver Papago {jihyung.moon, hyunchang.cho, lucypark}@navercorp.com Eunjeong L. Park Naver Papago Abstract Quality estimation (QE) is the task of auto- matically evaluating the quality of transla- tions without human-translated references. Calculating BLEU between the input sen- tence and round-trip translation (RTT) was once considered as a metric for QE, how- ever, it was found to be a poor predictor of translation quality. Recently, various pre- trained language models have made break- throughs in NLP tasks by providing se- mantically meaningful word and sentence embeddings. In this paper, we employ se- mantic embeddings to RTT-based QE. Our method achieves the highest correlations with human judgments, compared to previ- ous WMT 2019 quality estimation metric task submissions. While backward transla- tion models can be a drawback when using RTT, we observe that with semantic-level metrics, RTT-based QE is robust to the choice of the backward translation system. Additionally, the proposed method shows consistent performance for both SMT and NMT forward translation systems, imply- ing the method does not penalize a certain type of model. 1 Introduction A good machine translation (MT) system con- verts one language to another while preserving the meaning of a sentence. Given a pair of well- performing translation systems between two lan- guages, the meaning of a sentence should remain c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Input (en) ‘We know it won’t change students’ behaviour instantly. Reference (de) Wir wissen, dass es das Verhalten der Studenten nicht sofort ändern wird. Output (de) „Wir wissen, dass es das Verhalten der Schüler nicht sofort ändern wird. Round-trip (en) “We know that it will not change student behavior immediately. RTT-SENTBLEU: 14.99 (rank: 1947/1997) RTT-SBERT(*): 98.07 (rank: 1001/1997) RTT-BERTSCORE(*): 97.04 (rank: 1033/1997) Table 1: A sample of RTT-based evaluation methods with an example from the WMT19 English–German evaluation set. * denotes our proposed semantic-level methods (Detailed def- initions are described in Section 3). Note that SENTBLEU could not capture the similarity of the input and RTT. intact even after a round-trip translation (RTT) – the process of translating text from the source to target language (forward translation, FT) and translating the result back into the source language (backward translation, BT). If the MT systems work reasonably well and no human-produced ref- erence translations are provided, using RTT for translation evaluation seems like a natural choice. However, in the early 2000s, this practice was not recommended to be used as a translation evaluation method (Huang, 1990; Somers, 2005; van Zaanen and Zwarts, 2006). This argument was largely supported by the poor correlation be- tween BLEU (Papineni et al., 2002) for refer- ence and translated output and BLEU for input and RTT (We address this method again in Sec- tion 3 as RTT-BLEU). However, BLEU only mea- sures surface-level lexical similarity, thus penal- izing paraphrased sentences resulting from the round-trip translation as shown in Table 1. On the other …

Content truncated to 500 words