Unsupervised Bilingual Word Embedding Agreement for Unsupervised Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1235–1245 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1235 Unsupervised Bilingual Word Embedding Agreement for Unsupervised Neural Machine Translation Haipeng Sun1∗, Rui Wang2, Kehai Chen2, Masao Utiyama2, Eiichiro Sumita2, and Tiejun Zhao1 1Harbin Institute of Technology, Harbin, China 2National Institute of Information and Communications Technology (NICT), Kyoto, Japan hpsun@hit-mtlab.net, tjzhao@hit.edu.cn {wangrui, khchen, mutiyama, eiichiro.sumita}@nict.go.jp Abstract Unsupervised bilingual word embedding (UBWE), together with other technologies such as back-translation and denoising, has helped unsupervised neural machine translation (UNMT) achieve remarkable results in several language pairs. In previous methods, UBWE is first trained using non- parallel monolingual corpora and then this pre-trained UBWE is used to initialize the word embedding in the encoder and decoder of UNMT. That is, the training of UBWE and UNMT are separate. In this paper, we first empirically investigate the relationship between UBWE and UNMT. The empirical findings show that the performance of UNMT is significantly affected by the performance of UBWE. Thus, we propose two methods that train UNMT with UBWE agreement. Empirical results on several language pairs show that the proposed methods significantly outperform conventional UNMT. 1 Introduction Since 2013, neural network based bilingual word embedding (BWE) has been applied to several natural language processing tasks (Mikolov et al., 2013; Faruqui and Dyer, 2014; Xing et al., 2015; Dinu et al., 2015; Lu et al., 2015; Wang et al., 2016; Artetxe et al., 2016; Smith et al., 2017; Wang et al., 2018). Recently, researchers have found that supervision is not always necessary (Cao et al., 2016; Zhang et al., 2017). Several unsupervised BWE (UBWE) methods (Conneau et al., 2018; Artetxe et al., 2018a) have been proposed and these have achieved impressive performance in word- translation tasks. The success of UBWE makes unsupervised neural machine translation (UNMT) possible. The combination of UBWE with denoising autoencoder and back-translation has ∗Haipeng Sun was an internship research fellow at NICT when conducting this work. led to UNMT that relies solely on monolingual corpora, with remarkable results reported for several language pairs such as English-French and English-German (Artetxe et al., 2018c; Lample et al., 2018a). In previous methods, UBWE is first trained using non-parallel monolingual corpora. This pre- trained UBWE is then used to initialize the word embedding in the encoder and decoder of UNMT. That is, the training of UBWE and UNMT take place in separate steps. In this paper, we first empirically investigate the relationship between UBWE and UNMT. Our empirical results show that: • 1) There is a positive correlation between the quality of the pre-trained UBWE and the performance of UNMT. • 2) The UBWE quality significantly decreases during UNMT training. Based on these two findings, we hypothesize that the learning of UNMT with UBWE agreement would enhance UNMT performance. In detail, we propose two approaches, UBWE agreement regularization and UBWE adversarial training, to maintain the quality of UBWE during NMT training. Empirical results on several language pairs …

Content truncated to 500 words