The NiuTrans Machine Translation Systems for WMT22

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 366 - 374 December 7-8, 2022 ©2022 Association for Computational Linguistics The NiuTrans Machine Translation Systems for WMT22 Weiqiao Shan1, Zhiquan Cao1, Yuchen Han1, Siming Wu1, Yimin Hu1, Jie Wang1, Yi zhang1, Hang Cao1, Baoyu Hou1, Chenghao Gao1, Xiaowen Liu1, Tong Xiao1,2, Anxiang Ma1,2 and Jingbo Zhu1,2 1NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China shanweiqiao96@gmail.com , {xiaotong, maanxiang, zhujingbo}@mail.neu.edu.cn Abstract This paper describes the NiuTrans neural ma- chine translation systems of the WMT22 Gen- eral MT constrained task. We participate in four directions, including Chinese→English, English→Croatian, and Livonian↔English. Our models are based on several advanced Transformer variants, e.g., Transformer-ODE, Universal Multiscale Transformer (UMST). The main workflow consists of data filtering, large-scale data augmentation (i.e., iterative back-translation, iterative knowledge distilla- tion), and specific-domain fine-tuning. More- over, we try several multi-domain methods, such as a multi-domain model structure and a multi-domain data clustering method, to rise to this year’s newly proposed multi-domain test set challenge. For low-resource scenarios, we build a multi-language translation model to en- hance the performance and try to use the pre- trained language model (mBERT) to initialize the translation model. 1 Introduction We participate in the WMT22 General MT task, including Chinese→English (ZH→EN), English→Croatian (EN→HR), and Livonian↔English (LIV↔EN) in four directions. All of our systems are built with constrained data sets. We adopt some methods that have been proven to work well in WMT over the past few years (Li et al., 2019; Zhang et al., 2020; Zhou et al., 2021). At the same time, we also adopt some new model structures (Li et al., 2022; Jiang et al., 2020), data clustering (Aharoni and Goldberg, 2020), initialization (Guo et al., 2020), and training methods (Liu et al., 2021), which are described in detail below. For data preparation and augmentation, since fil- tering data could hurt the model performance on the general domain machine translation task, we apply several soft data filtering rules to preserve as much data as possible (Zhang et al., 2020; Zhou et al., 2021). To obtain the in-domain data, we use the open-source toolkit XenC (Rousseau, 2013) and specially try a domain clusters method based on the BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) model in the ZH→EN direction. We also use back-translation (Sennrich et al., 2016a) and knowledge distillation (Freitag et al., 2017) iter- atively to increase the size of in-domain data, which has been proved effective in recent years (Zhang et al., 2020; Zhou et al., 2021). onFor model architectures, our system is built several Transformer variants, including Transformer-RPR, Transformer-DLCL (Wang et al., 2019), Transformer-ODE (Li et al., 2021), Transformer-UMST (Li et al., 2022), and Transformer-based model with domain mix- ing (Jiang et al., 2020). We build a wide and deep model based on the pre-norm structure (Wang et al., 2019) and relative position representa- tion(RPR) (Shaw et al., 2018), inspired by the effec- tiveness of the deep model. Furthermore, …

Content truncated to 500 words