Alibaba’s Submission for the WMT 2020 APE Shared Task: Improving Automatic Post-Editing with Pre-trained Conditional Cross-Lingual BERT

Proceedings of the 5th Conference on Machine Translation (WMT), pages 789–796 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 789 Alibaba’s Submission for the WMT 2020 APE Shared Task: Improving Automatic Post-Editing with Pre-trained Conditional Cross-Lingual BERT Jiayi Wang∗, Ke Wang∗, Kai Fan, Yuqi Zhang, Jun Lu, Xin Ge, Yangbin Shi, Yu Zhao Alibaba Group Inc., Hangzhou, China {joanne.wjy,moyu.wk,k.fan,chenwei.zyq,joelu.luj, shiyi.gx,taiwu.syb}@alibaba-inc.com, kongyu@taobao.com Abstract The goal of Automatic Post-Editing (APE) is basically to examine the automatic methods for correcting translation errors generated by an unknown machine translation (MT) sys- tem. This paper describes Alibaba’s submis- sions to the WMT 2020 APE Shared Task for the English-German language pair. We design a two-stage training pipeline. First, a BERT-like cross-lingual language model is pre-trained by randomly masking target sen- tences alone. Then, an additional neural de- coder on the top of the pre-trained model is jointly fine-tuned for the APE task. We also ap- ply an imitation learning strategy to augment a reasonable amount of pseudo APE training data, potentially preventing the model to over- fit on the limited real training data and boost- ing the performance on held-out data. To ver- ify our proposed model and data augmentation, we examine our approach with the well-known benchmarking English-German dataset from the WMT 2017 APE task. The experiment re- sults demonstrate that our system significantly outperforms all other baselines and achieves the state-of-the-art performance. The final re- sults on the WMT 2020 test dataset show that our submission can achieve +5.56 BLEU and -4.57 TER with respect to the official MT base- line. 1 Introduction and Related Work Even machines can approach and achieve parity with human translations (Hassan et al., 2018) em- powered by a sequence-to-sequence fashion (Bah- danau et al., 2014; Vaswani et al., 2017), post- editing is still an important and necessary step in the translation process, especially in scenarios where extremely high-quality translation results are essentially required such as business legal doc- uments, technical product guides, medicine instruc- tions and so on. It is the process whereby humans ∗* indicates equal contribution. amend machine-generated translation to achieve an acceptable final product. Translation crowdsourc- ing paradigm, computer assisted translation (CAT) thus comes into being as demanded, which includes a hybrid of machine translation and human post- editing to meet translation scenarios with different quality requirements accordingly for accuracy, clar- ity, fluency, and domain adaptation. However, post-editing, while improving, that can match human understanding of meaning, nu- ance, tone, humor–the list goes on, it’s often worth paying extra more. The time spent on translation mistake corrections by humans remains substantial to the extent (L¨aubli et al., 2013) so that it even occasionally offsets the efficiency gained from the neural machine translation (NMT) systems. In this paper, we explore automatic post-editing (APE) in a deep learning framework where a two-stage train- ing pipeline is engaged. The goal of APE task is to examine automatic methods of correcting trans- lation mistakes produced by a black-box machine translation engine to improve the MT results. Hu- …

Content truncated to 500 words