WeChat Neural Machine Translation Systems for WMT21

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 243–254 November 10–11, 2021. ©2021 Association for Computational Linguistics 243 WeChat Neural Machine Translation Systems for WMT21 Xianfeng Zeng1∗, Yijin Liu12∗, Ernan Li1∗, Qiu Ran1∗, Fandong Meng1∗, Peng Li1, Jinan Xu2, and Jie Zhou1 1 Pattern Recognition Center, WeChat AI, Tencent Inc, China 2 Beijing Jiaotong University, Beijing, China {xianfzeng,yijinliu,cardli,soulcaptran,fandongmeng,patrickpli,withtomzhou}@tencent.com jaxu@bjtu.edu.cn Abstract This paper introduces WeChat AI’s participa- tion in WMT 2021 shared news translation task on English→Chinese, English→Japanese, Japanese→English and English→German. Our systems are based on the Transformer (Vaswani et al., 2017) with several novel and effective variants. In our experiments, we employ data filtering, large-scale synthetic data generation (i.e., back-translation, knowl- edge distillation, forward-translation, iterative in-domain knowledge transfer), advanced fine- tuning approaches, and boosted Self-BLEU based model ensemble. Our constrained systems achieve 36.9, 46.9, 27.8 and 31.3 case- sensitive BLEU scores on English→Chinese, English→Japanese, Japanese→English and English→German, respectively. The BLEU scores of English→Chinese, English→Japanese and Japanese→English are the highest among all submissions, and that of English→German is the highest among all constrained submissions. 1 Introduction We participate in the WMT 2021 shared news translation task in three language pairs and four language directions, English→Chinese, English↔Japanese, and English→German. In this year’s translation tasks, we mainly improve the fi- nal ensemble model’s performance by increasing the diversity of both the model architecture and the synthetic data, as well as optimizing the ensemble searching algorithm. Diversity is a metric we are particularly inter- ested in this year. To quantify the diversity among different models, we compute Self-BLEU (Zhu et al., 2018) from the translations of the models on the valid set. To be precise, we use the translation of one model as the hypothesis and the translations of other models as references to calculate an aver- ∗ Equal contribution. age BLEU score. A higher Self-BLEU means this model is less diverse. For model architectures (Vaswani et al., 2017; Meng and Zhang, 2019; Yan et al., 2020), we exploit several novel Transformer variants to strengthen model performance and diversity. Be- sides the Pre-Norm Transformer, the Post-Norm Transformer is also used as one of our baselines this year. We adopt some novel initialization meth- ods (Huang et al., 2020) to alleviate the gradi- ent vanishing problem of the Post-Norm Trans- former. We combine the Average Attention Trans- former (AAN) (Zhang et al., 2018) and Multi-Head- Attention (Vaswani et al., 2017) to derive a series of effective and diverse model variants. Furthermore, Talking-Heads Attention (Shazeer et al., 2020) is introduced to the Transformer and shows a signifi- cant diversity from all the other variants. For the synthetic data generation, we exploit the large-scale back-translation (Sennrich et al., 2016a) method to leverage the target-side monolingual data and the sequence-level knowledge distillation (Kim and Rush, 2016) to leverage the source-side of bilingual data. To use the source-side monolin- gual data, we explore forward-translation by en- semble models to get general domain synthetic data. We also use iterative in-domain knowledge transfer (Meng et al., 2020) to generate …

Content truncated to 500 words