Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3113–3124 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3113 Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models Takashi Wada1, Tomoharu Iwata2,3, and Yuji Matsumoto1,3 1Nara Institute of Science and Technology 2NTT Communication Science Laboratories 3RIKEN Center for Advanced Intelligence Project (AIP) 1{wada.takashi.wp7,matsu}@is.naist.jp 2tomoharu.iwata.gy@hco.ntt.co.jp Abstract Recently, a variety of unsupervised methods have been proposed that map pre-trained word embeddings of different languages into the same space without any parallel data. These methods aim to find a linear transformation based on the assumption that monolingual word embeddings are approximately isomor- phic between languages. However, it has been demonstrated that this assumption holds true only on specific conditions, and with limited resources, the performance of these methods decreases drastically. To overcome this prob- lem, we propose a new unsupervised multi- lingual embedding method that does not rely on such assumption and performs well under resource-poor scenarios, namely when only a small amount of monolingual data (i.e., 50k sentences) are available, or when the do- mains of monolingual data are different across languages. Our proposed model, which we call ‘Multilingual Neural Language Models’, shares some of the network parameters among multiple languages, and encodes sentences of multiple languages into the same space. The model jointly learns word embeddings of dif- ferent languages in the same space, and gen- erates multilingual embeddings without any parallel data or pre-training. Our experi- ments on word alignment tasks have demon- strated that, on the low-resource condition, our model substantially outperforms existing unsupervised and even supervised methods trained with 500 bilingual pairs of words. Our model also outperforms unsupervised meth- ods given different-domain corpora across lan- guages. Our code is publicly available1. 1 Introduction Learning cross-lingual or multilingual word em- bedding has been recognised as a very impor- tant research topic in natural language processing 1https://github.com/twadada/multilingual-nlm (NLP). Its objective is to map monolingual word embeddings of different languages into a com- mon space, and this research has been applied to many multilingual tasks such as machine transla- tion (Zou et al., 2013) and bilingual named entity recognition (Rudramurthy et al., 2016). It also en- ables the transfer of knowledge from one language into another (Xiao and Guo, 2014; Adams et al., 2017). A number of supervised and unsupervised methods have been proposed that obtain cross- lingual word embeddings. Both supervised and unsupervised methods aim to find such a linear transformation that maps word embeddings in a source language into a target language space. Su- pervised methods employ bilingual dictionaries to learn the mapping (Mikolov et al., 2013b; Xing et al., 2015; Smith et al., 2017; Artetxe et al., 2018a), while unsupervised ones utilise the sim- ilarities or distance of word embeddings spaces across different languages (Conneau et al., 2018; Zhang et al., 2017a; Xu et al., 2018; Artetxe et al., 2018b). Since the common objective of most of the supervised and unsupervised methods is to find an …

Content truncated to 500 words