SJTU-NICT’s Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 218–229 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 218 SJTU-NICT’s Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task Zuchao Li1,2,3, Hai Zhao1,2,3,∗, Rui Wang4,∗, Kehai Chen4, Masao Utiyama4, and Eiichiro Sumita4 1Department of Computer Science and Engineering, Shanghai Jiao Tong University (SJTU) 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institude, Shanghai Jiao Tong University, China 4National Institute of Information and Communications Technology (NICT), Kyoto, Japan charlee@sjtu.edu.cn, zhaohai@cs,sjtu.edu.cn, wangrui@nict.go.jp Abstract In this paper, we introduced our joint team SJTU-NICT ’s participation in the WMT 2020 machine translation shared task. In this shared task, we participated in four translation direc- tions of three language pairs: English-Chinese, English-Polish on supervised machine trans- lation track, German-Upper Sorbian on low- resource and unsupervised machine translation tracks. Based on different conditions of language pairs, we have experimented with diverse neural machine translation (NMT) techniques: document-enhanced NMT, XLM pre-trained language model enhanced NMT, bidirectional translation as a pre-training, ref- erence language based UNMT, data-dependent gaussian prior objective, and collaborative filtering self-training.BT-BLEU We also used the TF-IDF algorithm to filter the training set to obtain a domain more similar set with the test set for finetuning. In our submissions, the primary systems won the first place on English to Chinese, Polish to English, and German to Upper Sorbian translation directions. 1 Introduction Our SJTU-NICT team participated in the WMT20 shared task, including supervised track, unsu- pervised, and low-resource track. During the participation, we placed our attention on Polish (PL) → English (EN) and English (EN) → Chinese (ZH) on the supervised track, while on the unsupervised and low-resource track, the German ∗ Corresponding authors. This paper was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), Huawei-SJTU Long Term AI Project, Cutting- edge Machine Reading Comprehension and Language Model. Rui Wang was partially supported by JSPS grant-in-aid for early-career scientists (19K20354): “Unsupervised Neural Machine Translation in Universal Scenarios” and NICT tenure- track researcher startup fund “Toward Intelligent Machine Translation”. (DE) ↔ Upper Sorbian (HSB) both directions are focused. Our baseline system in supervised track is based on the Transformer big architecture proposed by Vaswani et al. (2017), in which its open- source implementation version Fairseq (Ott et al., 2019) is adopted. In the unsupervised and low-resource track, we draw on the successful experience of the XLM framework (Conneau et al., 2019), and used the two-stage training mode of masked language modeling (MLM) pre-training + back-translation (BT) finetune to obtain a very strong baseline performance. Marian (Junczys- Dowmunt et al., 2018) toolkit is utilized for training the decoder in reranking using machine translation targets instead of common GPT-style language modeling targets. In order to better play the role of WMT evaluation in polishing the methods proposed or improved …

Content truncated to 500 words