IIE’s Neural Machine Translation Systems for WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 300–304 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 300 IIE’s Neural Machine Translation Systems for WMT20 Xiangpeng Wei1,2, Ping Guo1,2, Yunpeng Li1,2, Xingsheng Zhang1,2, Luxi Xing1,2, Yue Hu1,2 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {weixiangpeng,huyue}@iie.ac.cn Abstract In this paper we introduce the systems IIE submitted for the WMT20 shared task on German↔French news translation. Our sys- tems are based on the Transformer architec- ture with some effective improvements. Mul- tiscale collaborative deep architecture, data se- lection, back translation, knowledge distilla- tion, domain adaptation, model ensemble and re-ranking are employed and proven effective in our experiments. Our German→French sys- tem achieved 35.0 BLEU and ranked the sec- ond among all anonymous submissions, and our French→German system achieved 36.6 BLEU and ranked the fourth in all anonymous submissions. 1 Introduction We participate in the WMT20 shared news translation task in one language pair and two language directions, German→French and French→German. Our methods are based on tech- niques and approaches used in submissions from past years (Deng et al., 2018; Ng et al., 2019; Sun et al., 2019; Li et al., 2019; Xia et al., 2019), in- cluding the use of subword models (Sennrich et al., 2016), iterative back-translation, knowledge distil- lation, model ensembling and several techniques we proposed recently (Wei et al., 2020b,a). For our submissions of two language directions, we adopt the deep transformer architectures (48- layer) based on multiscale collaboration mecha- nism (Wei et al., 2020b) as our baseline, which outperformed the standard Transformer-Big as well as shallower models significantly in terms of translation quality. We also use an iterative back-translation approach (Zhang et al., 2018) with the controllable sampling to extend the back trans- lation method by jointly training source-to-target and target-to-source NMT models. Moreover, the knowledge distillation (Freitag et al., 2017) is em- ployed to leverage the source-side monolingual data. For our final models, we apply a domain- specific fine-tuning process and model ensembling, and decode using noisy channel model re-ranking. The paper is structured as follows: Section 2 describes the techniques we used, then section 3 shows the experimental settings and results. Fi- nally, we conclude our work in Section 4. 2 Our Techniques 2.1 Multiscale Collaborative Deep Models The structure of NMT models has evolved quickly, such as RNN-based (Wu et al., 2016), CNN-based (Gehring et al., 2017) and attention- based (Vaswani et al., 2017) systems. Deep neural networks have revolutionized the state-of-the-art in various communities, from computer vision to nat- ural language processing. We adopt the deep trans- former model proposed by our work (Wei et al., 2020b). Instead of relying on the whole encoder stack to directly learn a desired representation, we let each encoder block learn a fine-grained represen- tation and enhance it by encoding spatial dependen- cies using a bottom-up network. For coordination, we attend each block of the decoder to both the corresponding …

Content truncated to 500 words