A Simple Approach to Learning Unsupervised Multilingual Embeddings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2995–3001, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2995 A Simple Approach to Learning Unsupervised Multilingual Embeddings Pratik Jawanpuria, Mayank Meghwanshi, Bamdev Mishra Microsoft, India {pratik.jawanpuria,mamegh,bamdevm}@microsoft.com Abstract Recent progress on unsupervised cross-lingual embeddings in the bilingual setting has given the impetus to learning a shared embedding space for several languages. A popular frame- work to solve the latter problem is to solve the following two sub-problems jointly: 1) learn- ing unsupervised word alignment between sev- eral language pairs, and 2) learning how to map the monolingual embeddings of every lan- guage to shared multilingual space. In con- trast, we propose a simple approach by decou- pling the above two sub-problems and solv- ing them separately, one after another, using existing techniques. We show that this pro- posed approach obtains surprisingly good per- formance in tasks such as bilingual lexicon in- duction, cross-lingual word similarity, multi- lingual document classification, and multilin- gual dependency parsing. When distant lan- guages are involved, the proposed approach shows robust behavior and outperforms ex- isting unsupervised multilingual word embed- ding approaches. 1 Introduction Learning cross-lingual word representations has been the focus of many recent works (Mikolov et al., 2013; Faruqui and Dyer, 2014; Artetxe et al., 2016). It aims at learning a shared embedding space for words across two (bilingual word embed- ding) or more languages (multilingual word em- bedding or MWE), by mapping similar words (or concepts) across different languages close to each other in a shared embedding space. Such a repre- sentation is useful in various applications such as cross-lingual text classification (Klementiev et al., 2012), building bilingual lexicons (Mikolov et al., 2013), cross-lingual information retrieval (Vuli´c and Moens, 2015), and machine translation (Gu et al., 2018), to name a few. Mikolov et al. (2013) showed that the geometric arrangement of word embeddings could be (ap- proximately) preserved by linearly transforming the word embeddings from one language space to another. Subsequently, several works have ex- plored learning bilingual word embeddings in both supervised (Xing et al., 2015; Artetxe et al., 2016, 2018a; Smith et al., 2017; Jawanpuria et al., 2019) and unsupervised (Zhang et al., 2017a,b; Conneau et al., 2018; Artetxe et al., 2018b; Alvarez-Melis and Jaakkola, 2018; Hoshen and Wolf, 2018; Grave et al., 2019; Jawanpuria et al., 2020a) settings. Representing word embeddings of many lan- guages in a common shared space is desirable to allow knowledge transfer between different lan- guages. Chen and Cardie (2018) are among the first to propose unsupervised learning of MWEs. They extend the GAN-based iterative refinement procedure for learning bilingual word embeddings (Conneau et al., 2018) to the multilingual setting. However, adversarial training has known concerns of optimization stability with distant language pairs (Søgaard et al., 2018). Alaux et al. (2019) propose a joint optimization framework for learning bilin- gual lexicons and mappings between several pairs of languages. They obtain the bilingual lexicons using the Gromov-Wasserstein approach (Alvarez- Melis and Jaakkola, 2018) and …

Content truncated to 500 words