The NiuTrans Machine Translation Systems for WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 338–345 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 338 The NiuTrans Machine Translation Systems for WMT20 Yuhao Zhang1, Ziyang Wang1, Runzhe Cao1, Binghao Wei1, Weiqiao Shan1, Shuhan Zhou1, Abudurexiti Reheman1, Tao Zhou1, Xin Zeng1, Laohu Wang1, Xiaoqian Liu1, Xunjuan Zhou1, Yongyu Mu1, Jingnan Zhang1, Yinqiao Li1, Bei Li1, Tong Xiao1,2 and Jingbo Zhu1,2 1NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China yoohao.zhang@gmail.com, wangziyang@stumail.neu.edu.cn {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract This paper describes NiuTrans neural ma- chine translation systems of the WMT20 news translation tasks. We participated in Japanese↔English, English→Chinese, Inuktitut→English and Tamil→English total five tasks and rank first in Japanese↔English both sides. We mainly utilized iterative back- translation, different depth and widen model architectures, iterative knowledge distillation and iterative fine-tuning. And we find that adequately widened and deepened the model simultaneously, the performance will signif- icantly improve. Also, iterative fine-tuning strategy we implemented is effective during adapting domain. For Inuktitut→English and Tamil→English tasks, we built multilingual models separately and employed pretraining word embedding to obtain better performance. 1 Introduction This paper describes the NiuTrans submis- sions to the WMT20 news tasks, including English→Chinese (EN→ZH), Tamil→English (TA→EN), Inuktitut→English (IU→EN) and Japanese↔English (JA↔EN) five directions and all of our systems were built with constrained data sets. Some useful methods in the WMT18 (Wang et al., 2018) and WMT19 (Li et al., 2019) sub- missions are also reused this time, such as model ensemble, knowledge distillation (KD) et al., and we explore some novel approaches this year. For this participation, we experimented with some deeper and wider Transformer (Vaswani et al., 2017) architectures to get reliable baselines, nu- cleus sampling (Holtzman et al., 2020) in back- translation to generate more suitable pseudo bilin- gual sentences, more effectively fine-tuning strat- egy to adapt domain. Particularly in the low- resources tasks, {TA,IU}→EN, we built multilin- gual neural machine translation by using some sim- ilar language to get better performance and further replaced decoder’s word embedding by an English pretraining Transformer language model’s which trained by two monolingual in-domain data cor- pora. Furthermore, we presented a new fine-tuning pat- tern which could significantly improve the BLEU score on the test set, and it worked well on all five tasks whether it is a low or rich resource. We care- fully rethought this strategy and found the main gain came from domain adaptation and improved inferior translations. Our systems and this paper followed six main steps:1) data preprocessing and filter, 2) iterative back-translation to generate pseudo bilingual data, 3) using different model architectures to enhance the diversity of translation, 4) iterative knowledge distillation by in-domain monolingual data, 5) itera- tive fine-tuning with in-domain using small training batch, 6) translation post-process. 2 System Overview 2.1 Data Preprocessing and Filtering For EN→ZH and JA↔EN tasks, we first normal- ized the punctuation in Chinese and Japanese mono- lingual data by using Moses (Koehn et al., 2007) normalize-punctuation.perl script. En- glish and Inuktitut sentences were …

Content truncated to 500 words