Cross-Lingual Transformers for Neural Automatic Post-Editing

Proceedings of the 5th Conference on Machine Translation (WMT), pages 772–776 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 772 Cross-Lingual Transformers for Neural Automatic Post-Editing Dongjun Lee Bering Lab, Republic of Korea djlee@beringlab.com Abstract In this paper, we describe the Bering Lab’s submission to the WMT 2020 Shared Task on Automatic Post-Editing (APE). First, we propose a cross-lingual Transformer architec- ture that takes a concatenation of a source sen- tence and a machine-translated (MT) sentence as an input to generate the post-edited (PE) out- put. For further improvement, we mask incor- rect or missing words in the PE output based on word-level quality estimation and then pre- dict the actual word for each mask based on the fine-tuned cross-lingual language model (XLM-RoBERTa). Finally, to address the over- correction problem, we select the final out- put among the PE outputs and the original MT sentence based on a sentence-level qual- ity estimation. When evaluated on the WMT 2020 English-German APE test dataset, our system improves the NMT output by −3.95 and +4.50 in terms of TER and BLEU, respec- tively. 1 Introduction Automatic post-editing (APE) is the task of auto- matically correcting errors in the output of a ma- chine translation (MT) system by learning from hu- man corrections (Chatterjee et al., 2019). APE can be viewed as a cross-lingual sequence-to-sequence task, which takes a source sentence and the corre- sponding MT output as inputs and generates the post-edited (PE) output. Our work is inspired by XLM-RoBERTa (XLM- R) (Conneau et al., 2019), a cross-lingual lan- guage model, which shows the state-of-the-art per- formance for a wide range of cross-lingual tasks. XLM-R takes a concatenation of two sentences in different languages as an input to generate cross- lingual representations. Similarly, we propose a Transformer (Vaswani et al., 2017) architecture for APE in which the encoder uses the same architec- ture as XLM-R. In addition, we use XLM-R-based translation quality estimation (QE) (Lee, 2020) to further im- prove the PE output of the Transformer. QE is the task of estimating the quality of the MT out- put when only the source text is provided (Fonseca et al., 2019). We use two granularity levels of QE: word-level and sentence-level. Based on the word- level QE, we try to correct the wrong words or insert the missing words in the PE output. Through the sentence-level QE, we select the best translation among PE outputs and the original MT sentence to prevent over-correction (i.e., one of the APE models rephrases an already correct MT output). Our contributions are summarized as follows: • We propose a Transformer (Vaswani et al., 2017) architecture for APE in which an en- coder takes concatenation of a source and MT sentence as an input to generate a cross- lingual representation and a decoder generates a PE output. • We incorporate a word-level QE-based word masking. We replace BAD words with <mask> token or insert <mask> token for missing words in the PE output of the Trans- former based …

Content truncated to 500 words