Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3525–3535 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3525 Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation Haipeng Sun1∗, Rui Wang2, Kehai Chen2, Masao Utiyama2, Eiichiro Sumita2, and Tiejun Zhao1 1Harbin Institute of Technology, Harbin, China 2National Institute of Information and Communications Technology (NICT), Kyoto, Japan hpsun@hit-mtlab.net, tjzhao@hit.edu.cn {wangrui, khchen, mutiyama, eiichiro.sumita}@nict.go.jp Abstract Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs. However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. That is, research on multilingual UNMT has been limited. In this paper, we empirically in- troduce a simple method to translate between thirteen languages using a single encoder and a single decoder, making use of multilingual data to improve UNMT for all language pairs. On the basis of the empirical findings, we propose two knowledge distillation methods to further enhance multilingual UNMT per- formance. Our experiments on a dataset with English translated to and from twelve other languages (including three language families and six language branches) show remarkable results, surpassing strong unsupervised indi- vidual baselines while achieving promising performance between non-English language pairs in zero-shot translation scenarios and alleviating poor performance in low-resource language pairs. 1 Introduction Recently, neural machine translation (NMT) has been adapted to the unsupervised scenario in which NMT is trained without any bilingual data. Un- supervised NMT (UNMT) (Artetxe et al., 2018; Lample et al., 2018a) requires only monolingual corpora. UNMT achieves remarkable results by us- ing a combination of diverse mechanisms (Lample et al., 2018b) such as an initialization with bilingual word embeddings, denoising auto-encoder (Vin- cent et al., 2010), back-translation (Sennrich et al., 2016a), and shared latent representation. More re- cently, Lample and Conneau (2019) achieves better ∗Haipeng Sun was an internship research fellow at NICT when conducting this work. UNMT performance by introducing the pretrained language model. However, conventional UNMT can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time (Wang et al., 2020). Multilingual UNMT (MUNMT) translating mul- tiple languages at the same time can save substan- tial training time and resources. Moreover, the performance of MUNMT in similar languages can promote each other. Research on MUNMT has been limited and there are only a few pioneer stud- ies. For example, Xu et al. (2019) and Sen et al. (2019) proposed a multilingual scheme that jointly trains multiple languages with multiple decoders. However, the performance of their MUNMT is much worse than our re-implemented individual baselines (shown in Tables 2 and 3) and the scale of their study is modest (i.e., 4-5 languages). In this paper, we empirically introduce an unified framework to translate among thirteen languages (including three language families and six language branches) using a single encoder and single de- coder, making use of multilingual data to improve UNMT for all languages. On …

Content truncated to 500 words