WeChat Neural Machine Translation Systems for WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 239–247 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 239 WeChat Neural Machine Translation Systems for WMT20 Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xianfeng Zeng, Qinsong Zeng, Peng Li, Ming Chen, Jie Zhou, Sifan Liu and Hao Zhou WeChat AI, Tencent, China {fandongmeng,elliottyan,yijinliu,masongao,xianfzeng}@tencent.com {qinzzeng,patrickpli,mingchen,withtomzhou,stephenliu,harveyzhou}@tencent.com Abstract We participate in the WMT 2020 shared news translation task on Chinese→English. Our sys- tem is based on the Transformer (Vaswani et al., 2017) with effective variants and the DTMT (Meng and Zhang, 2019) architec- ture. In our experiments, we employ data selection, several synthetic data generation approaches (i.e., back-translation, knowledge distillation, and iterative in-domain knowledge transfer), advanced finetuning approaches and self-bleu based model ensemble. Our con- strained Chinese→English system achieves 36.9 case-sensitive BLEU score, which is the highest among all submissions. 1 Introduction Our WeChat AI team participates in the WMT 2020 shared news translation task on Chinese→English. In this year’s translation task, we mainly focus on exploiting several effective model architectures, better data augmentation, training and model en- semble strategies. For model architectures, we mainly exploit two different architectures in our approaches, namely Transformers and RNMT. For Transformers, we im- plement the Deeper transformer with Pre-Norm, the Wider Transformer with a larger filter-size and the average attention based transformer (Zhang et al., 2018). For the RNMT, we use the deep transition based DTMT (Meng and Zhang, 2019) model. We finally ensemble four kinds of models in our sys- tem. For synthetic data generation, we explore various methods for out-of-domain and in-domain data gen- eration. For out-of-domain data generation, we ex- plore the back-translation method (Sennrich et al., 2016a) to leverage the target side monolingual data and the knowledge distillation method (Kim and Rush, 2016) to leverage the source side of golden parallel data. For in-domain data generation, we employ iterative in-domain knowledge transfer to leverage the source-side monolingual data and golden parallel data. Furthermore, data augmenta- tion methods, including noisy fake data (Wu et al., 2019) and sampling (Edunov et al., 2018), are used for training more robust NMT models. For training strategies, we mainly focus on the parallel scheduled sampling (Mihaylova and Mar- tins, 2019; Duckworth et al., 2019), the target de- noising and minimum risk training (Shen et al., 2016; Wang and Sennrich, 2020) algorithm for in- domain finetuning. We also exploit a self-bleu (Zhu et al., 2018) based model ensemble approach to enhance our sys- tem. As a result, our constrained Chinese→English system achieves the highest case-sensitive BLEU score among all submitted systems. In the remainder of this paper, we start with an overview of model architectures in Section 2. Sec- tion 3 describes the details of our systems and train- ing strategies. Then Section 4 shows the experimen- tal settings and results. Finally, we conclude our work in Section 5. 2 Model Architectures In this section, we first describe the model archi- tectures we use in the Chinese−→English Shared Task, including the Transformer-based (Vaswani et al., …

Content truncated to 500 words