The Volctrans Machine Translation System for WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 305–312 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 305 The Volctrans Machine Translation System for WMT20 Liwei Wu1, Xiao Pan1, Zehui Lin2∗, Yaoming Zhu3†, Mingxuan Wang1, Lei Li1 1ByteDance AI Lab, Beijing, China {wuliwei.000, panxiao.94, wangmingxuan.89, lilei.lab}@bytedance.com 2Fudan University, Shanghai China linzh18@fudan.edu.cn 3Shanghai Jiao Tong University, Shanghai China ymZhu@apex.sjtu.edu.cn Abstract This paper describes our VolcTrans system on WMT20 shared news translation task. We par- ticipated in 14 translation directions. Our basic systems are based on Transformer (Vaswani et al., 2017), with several variants (wider or deeper Transformers, dynamic convolutions). The final system includes text pre-process, data selection, synthetic data generation, ad- vanced model ensemble, and multilingual pre- training. 1 Introduction We participated in the WMT2020 shared news translation task in 14 directions: English↔Chinese, English↔German, French↔German, English↔Polish, English↔Tamil,English↔Pashto,English↔Khmer, covering language pairs from high to low resources. In this year’s translation task, we mainly focus on exploiting self-supervised and unsupervised meth- ods for NMT to make full use of the monolingual data (Lin et al., 2020; Yang et al., 2019). We aims at building a general training frame- work which can be well applied to different trans- lation directions. Our models are mainly based on the Transformer (Vaswani et al., 2017). Tech- niques used in the submitted systems include iter- ative back-translation, knowledge distillation. We also employed several tricks to improve in-domain BLEU scores, typically in-domain transfer learn- ing. We also experimented with a multilingual pre- training technique which we proposed recently (Lin et al., 2020). 2 Baseline Models We apply two different NMT skeletons for the shared news translation as our baseline systems. ∗Intern at ByteDance †Intern at ByteDance We use the implementations in Fairseq(Ott et al., 2019). All models are trained with Adam optimizer (Kingma and Ba, 2014). We use the “inverse sqrt lr” scheduler with 4000 warm-up steps and set the max learning rate to 5e-4. The betas are (0.9, 0.98). During training, the batches are made of similar length sequences, so we avoid extreme cases where most sequences in the batch are short and we are required to add lots of pad tokens to each of them because one sequence of the same batch is very long. We limit the batch size to 8192 tokens per GPU, to avoid running out of GPU memory. Mean- while, to achieve a larger batch size to improve the performance(Ott et al., 2018), we set the parameter “update frequency” to 8, and train the model on 8 GPUs, resulting in an actual batch token size = 8192 × 8 × 8. During training, we employ label smoothing of 0.1 and set dropout rate (Hinton et al., 2012) to 0.2. 2.1 Transformer Following Sun et al. (2019); Wang et al. (2018), we use different architectures for Trans- former(Vaswani et al., 2017) to increase the model diversity and potentially get a better ensemble model. • Transformer 15e6d: According to Sun et al. (2019), a transformer with larger encoder layer number can learn …

Content truncated to 500 words