Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3975–3981 June 6–11, 2021. ©2021 Association for Computational Linguistics 3975 Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios Haipeng Sun1,2∗, Rui Wang3, Kehai Chen4, Masao Utiyama4, Eiichiro Sumita4, and Tiejun Zhao1† 1Harbin Institute of Technology, Harbin, China 2JD AI Research, Beijing, China 3Shanghai Jiao Tong University, Shanghai, China 4National Institute of Information and Communications Technology (NICT), Kyoto, Japan sunhaipeng6@jd.com, wangrui.nlp@gmail.com, tjzhao@hit.edu.cn {khchen, mutiyama, eiichiro.sumita}@nict.go.jp Abstract Unsupervised neural machine translation (UNMT) that relies solely on massive mono- lingual corpora has achieved remarkable results in several translation tasks. However, in real-world scenarios, massive monolingual corpora do not exist for some extremely low-resource languages such as Estonian, and UNMT systems usually perform poorly when there is not adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose UNMT self-training mechanisms to train a robust UNMT system and improve its performance in this case. Experimental results on several language pairs show that the proposed methods substantially outperform conventional UNMT systems. 1 Introduction Recently, unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has attracted a high level of interest in the machine translation community (Artetxe et al., 2018; Lample et al., 2018a; Yang et al., 2018; Lam- ple et al., 2018b; Wu et al., 2019; Sun et al., 2019, 2020b). With the help of cross-lingual language model pretraining (Lample and Conneau, 2019; Song et al., 2019; Sun et al., 2020a), the denois- ing auto-encoder (Vincent et al., 2010), and back- translation (Sennrich et al., 2016a), UNMT has achieved remarkable results in several translation tasks. However, in real-world scenarios, in contrast to the many large corpora available for high-resource languages such as English and French, massive monolingual corpora do not exist for some ex- tremely low-resource languages such as Estonian. ∗Part of this work was done when Haipeng Sun and Rui Wang were an internship research fellow and a researcher at NICT, respectively. † Corresponding author. Data size (sentences) En-Fr Fr-En 50M En and 50M Fr (Baseline) 36.63 34.38 25M En and 25M Fr 36.59 34.34 50M En and 2M Fr 31.01 31.06 2M En and 50M Fr 31.84 30.21 2M En and 2M Fr 30.91 29.86 Table 1: UNMT performance (BLEU score) for differ- ent training data sizes on En–Fr language pairs. The UNMT system usually performs poorly in a low-resource scenario when there is not an ade- quate training corpus for one language. In this paper, we first define and analyze the un- balanced training data scenario for UNMT. Based on this scenario, we propose a self-training mech- anism for UNMT. In detail, we propose self- training with unsupervised training (ST-UT) and self-training with pseudo-supervised training (ST- PT) strategies to train a robust UNMT system that performs better in this scenario. To the best of our knowledge, this paper is …

Content truncated to 500 words