PATQUEST: Papago Translation Quality Estimation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 991–998 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 991 PATQUEST: Papago Translation Quality Estimation Yujin Baek∗,† Graduate School of AI, KAIST yujinbaek@kaist.ac.kr Zae Myung Kim∗ Papago, Naver Corp. zaemyung.kim@navercorp.com Jihyung Moon Papago, Naver Corp. jihyung.moon@navercorp.com Hyunjoong Kim Papago, Naver Corp. soy.lovit@navercorp.com Eunjeong L. Park Papago, Naver Corp. lucy.park@navercorp.com Abstract This paper describes the system submitted by Papago team for the quality estimation task at WMT 2020. It proposes two key strategies for quality estimation: (1) task-specific pretrain- ing scheme, and (2) task-specific data augmen- tation. The former focuses on devising learn- ing signals for pretraining that are closely re- lated to the downstream task. We also present data augmentation techniques that simulate the varying levels of errors that the downstream dataset may contain. Thus, our PATQUEST models are exposed to erroneous translations in both stages of task-specific pretraining and finetuning, effectively enhancing their gener- alization capability. Our submitted models achieve significant improvement over the base- lines for Task 1 (Sentence-Level Direct Assess- ment; EN-DE only), and Task 3 (Document- Level Score). 1 Introduction With the widespread use of machine translation sys- tems, there is a growing need to evaluate translated results at low-cost. The task of quality estima- tion (QE) addresses this issue, where the quality of a translation is predicted automatically given the source sentence and its translation. The estimated quality can inform users about the reliability of the translation, or whether it needs to be post-edited. Previous QE systems generally include pretrain- ing and finetuning steps, where the former step involves masked language modeling (MLM) uti- lizing large parallel corpora, with the expectation that the models will learn cross-lingual relation- ships (Kepler et al., 2019; Kim et al., 2019). The models are, in turn, finetuned with task-specific data. However, while the pretraining step involves ∗Equal contribution † Work done during internship at Naver Corp. Task-specific pretraining Task-specific finetuning Task1 (Section 2) - Omitted word - Word order - Lexical selection - Repeated phrase - SentenceBERT - BERTScore - GPT-2 (ex: BERT, XLM) Task3 (Section 3) - Omitted word (preposition, determiner) - Word order - Lexical selection - Wrong preposition - Pseudo-MQM Perturbations Label Pretrained model Task-specific training Figure 1: Overview of our approach for Task 1 and 3. training data with near-perfect translations, low- quality translations are only introduced during the finetuning step. In this work, we suggest two key strategies that could alleviate this pretrain-finetune discrepancy in QE tasks by: (1) adopting a task-specific pre- training objective which is close to that of the downstream task, and (2) generating abundant task- specific erroneous sentence pairs and their learning signals. Our approach, which is depicted in Fig- ure 1, is motivated from BLEURT (Sellam et al., 2020), where we extend their general approach to the bilingual QE setting. Our submitted systems achieve significant improvements in performance over the baseline systems on WMT20 Shared Tasks for QE (Specia et al., 2020): an absolute gain of +35.2% in …

Content truncated to 500 words