Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training

Proceedings of the 28th International Conference on Computational Linguistics, pages 4239–4250 Barcelona, Spain (Online), December 8-13, 2020 4239 Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training Haipeng Sun1∗, Rui Wang2, Kehai Chen2, Xugang Lu2, Masao Utiyama2, Eiichiro Sumita2, and Tiejun Zhao1† 1Harbin Institute of Technology, Harbin, China 2National Institute of Information and Communications Technology (NICT), Kyoto, Japan sunhaipeng.nlp@gmail.com, tjzhao@hit.edu.cn {wangrui,khchen,xugang.lu,mutiyama,eiichiro.sumita}@nict.go.jp Abstract Unsupervised neural machine translation (UNMT) has recently attracted great interest in the ma- chine translation community. The main advantage of the UNMT lies in its easy collection of required large training text sentences while with only a slightly worse performance than super- vised neural machine translation which requires expensive annotated translation pairs on some translation tasks. In most studies, the UMNT is trained with clean data without considering its robustness to the noisy data. However, in real-world scenarios, there usually exists noise in the collected input sentences which degrades the performance of the translation system since the UNMT is sensitive to the small perturbations of the input sentences. In this paper, we first time explicitly take the noisy data into consideration to improve the robustness of the UNMT based systems. First of all, we clearly defined two types of noises in training sentences, i.e., word noise and word order noise, and empirically investigate its effect in the UNMT, then we propose adversarial training methods with denoising process in the UNMT. Experimental results on sev- eral language pairs show that our proposed methods substantially improved the robustness of the conventional UNMT systems in noisy scenarios. 1 Introduction Recently, unsupervised neural machine translation (UNMT) has attracted great interest in the machine translation community (Artetxe et al., 2018; Lample et al., 2018a; Yang et al., 2018; Lample et al., 2018b; Sun et al., 2019; Sun et al., 2020b). Typically, UNMT relies solely on monolingual corpora rather than bilingual parallel data in supervised neural machine translation (SNMT) to model transla- tions between the source language and target language and has achieved remarkable results on several translation tasks (Conneau and Lample, 2019). However, previous work only focus on how to build state- of-the-art UNMT systems on the clean data and ignore the robustness of UNMT on the noisy data. In the real-world scenario, there often exists noises or perturbations in the input sentences, for example, word character misspelling, replacement, or word position misordering, etc. The translation model is sensitive to these perturbations, leading to various errors even the perturbations are small. The existing neural translation system, which lacks of robustness, is difficult to be widely applied to the noisy-data scenario (denoted as noisy scenario in the following sections). Therefore, the robustness of neural translation system is not only worthy of being studied, but also very essential in the real-world scenarios. The robustness of SNMT (Belinkov and Bisk, 2018; Cheng et al., 2018; Cheng et al., 2019; Karpukhin et al., 2019) has been well-studied. However, most previous work only focus on the effect of the word substitution for translation performance, and ignore the effect …

Content truncated to 500 words