The NiuTrans Machine Translation Systems for WMT21

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 265–272 November 10–11, 2021. ©2021 Association for Computational Linguistics 265 The NiuTrans Machine Translation Systems for WMT21 Shuhan Zhou1, Tao Zhou1, Binghao Wei1, Yingfeng Luo1, Yongyu Mu1, Zefan Zhou1, Chenglong Wang1, Xuanjun Zhou1, Chuanhao Lv1, Yi Jing1, Laohu Wang1, Jingnan Zhang1, Canan Huang1, Zhongxiang Yan1, Chi Hu1, Bei Li1, Tong Xiao1,2 and Jingbo Zhu1,2 1NLP Lab, School of Computer Science and Engineering, Northeastern University 2NiuTrans Research, Shenyang, China zhoushuhan199710@163.com,zhoutao_neu@outlook.com {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract This paper describes NiuTrans neural ma- chine translation systems of the WMT 2021 news translation tasks. We made sub- missions to 9 language directions, includ- ing English↔{Chinese, Japanese, Russian, Icelandic} and English→Hausa tasks. Our primary systems are built on several effec- tive variants of Transformer, e.g., Transformer- DLCL, ODE-Transformer. We also utilize back-translation, knowledge distillation, post- ensemble, and iterative fine-tuning techniques to enhance the model performance further. 1 Introduction Our NiuTrans team participated in the WMT 2021 news translation shared tasks, including English↔Chinese (EN↔ZH), English↔Japanese (EN↔JA), English↔Russian (EN↔RU), English↔Icelandic (EN↔IS) and English→Hausa (EN→HA), nine submissions in total. All of our systems were built with constrained data sets. We adopt some effective models and useful methods, which have been witnessed the success in previous papers (Wang et al., 2018; Li et al., 2019; Zhang et al., 2020; Meng et al., 2020; Wu et al., 2020b; Chen et al., 2020; Yu et al., 2020; Wu et al., 2020a; Wei et al., 2020). To enhance the performance of the single model, we choose pre-normalized Transformer-DLCL (Wang et al., 2019) and ODE-Transformer (Li et al., 2021a) as the backbone. All systems are built upon the relative position representation (Shaw et al., 2018) due to its strong performance when models are deep (Li et al., 2020). For the system combi- nation, we adopt the post-ensemble (Kobayashi, 2018) to find the most similar hypothesis among several ensemble outputs, which could be regarded as a reranking technique without pre-training. Pre- vious works have emphasized the importance of diversity when building ensemble systems. Be- sides the architecture diversity, we also adopt iter- ative ensemble knowledge distillation leveraging the source-side monolingual data to enlarge the diversity. More details please refer to (Li et al., 2019). Our data preparation pipeline consists of three- fold: (i) For the data filtering. We use a stricter cleaning process than last year (Zhang et al., 2020). Details will be discussed in Section 2.1. (ii) For the data augmentation, both iterative back- translation (Sennrich et al., 2016a) method, and it- erative knowledge distillation (Freitag et al., 2017) method are employed to take the full advantage of monolingual data provided by the WMT orga- nization. In the back-translation stage, we lever- age target-side monolingual sentences to generate source-side pseudo sentences and use a nucleus sampling (Holtzman et al., 2019) decoding strategy to improve the generalization ability. Furthermore, we leverage in-domain source-side monolingual data by applying iterative knowledge distillation. (iii) For data selection, it’s hard to find massive in-domain data for low-resource languages to train a …

Content truncated to 500 words