POSTECH-ETRI’s Submission to the WMT2020 APE Shared Task: Automatic Post-Editing with Cross-lingual Language Model

Proceedings of the 5th Conference on Machine Translation (WMT), pages 777–782 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 777 POSTECH-ETRI’s Submission to the WMT2020 APE Shared Task: Automatic Post-Editing with Cross-lingual Language Model Jihyung Lee1, WonKee Lee1, Jaehun Shin1 Baikjin Jung1, Young-Kil Kim3, Jong-Hyeok Lee1,2 1Departmet of Computer Science and Engineering, 2Graduate School of Artificial Intelligence, Pohang University of Science and Technology (POSTECH), Republic of Korea 3Electronics and Telecommunications Research Institute, Republic of Korea {1jihyung.lee, 1wklee, 1jaehun.shin, 1bjjung, 1,2jhlee}@postech.ac.kr 3kimyk@etri.re.kr Abstract This paper describes POSTECH-ETRI’s sub- mission to WMT2020 for the shared task on automatic post-editing (APE) for 2 language pairs: English–German (En–De) and English– Chinese (En–Zh). We propose APE sys- tems based on a cross-lingual language model, which jointly adopts translation language mod- eling (TLM) and masked language modeling (MLM) training objectives in the pre-training stage; the APE models then utilize jointly learned language representations between the source language and the target language. In addition, we created 19 million new sythetic triplets as additional training data for our fi- nal ensemble model. According to experi- mental results on the WMT2020 APE develop- ment data set, our models showed an improve- ment over the baseline by TER of −3.58 and a BLEU score of +5.3 for the En–De subtask; and TER of −5.29 and a BLEU score of +7.32 for the En–Zh subtask. 1 Introduction Automatic post-editing (APE) is a subtask of MT, which aims to improve MT outputs by directly modifying machine-translated sentences (Chatter- jee et al., 2019). Using APE systems to correct such errors that are automatically detectable can greatly reduce human effort compared to correct- ing machine-translated sentences manually from scratch (Pal et al., 2016). Given that neural-network systems require a large quantity of training data, creating APE triplets, which each consist of a source sentence (src), a machine-translated sentence (mt), and a manually post-edited sentence (pe), requires a lot of human labor. Furthermore, because neural APE is a recently minted field of study, only a few small- sized training data sets are available at present. To mitigate such data shortage, several methods are proposed such that 1) create artificial APE triplets (Junczys-Dowmunt and Grundkiewicz, 2016; Ne- gri et al., 2018); and 2) apply ‘transfer learning’ (Correia and Martins, 2019; Lopes et al., 2019). We believe that pre-trained models such as ELMo (Peters et al., 2018), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2019) helped APE models learn rich language representations that compensated for the performance loss caused by using an insufficient quantity of training data. APE is a task that handles both src and mt si- multaneously, and learning a joint representation of these two inputs requires an understanding of both languages. Although previous works that used BERT have shown that transfer learning is effective in APE (Correia and Martins, 2019; Lopes et al., 2019), adopting BERT as a pre-trained language model may restrict to properly model the relation between two different languages because BERT is trained only on monolingual …

Content truncated to 500 words