Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach

Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach Pratik Jawanpuria1, Arjun Balgovind2∗, Anoop Kunchukuttan1, Bamdev Mishra1 1Microsoft, India 2IIT Madras, India 1{pratik.jawanpuria,ankunchu,bamdevm}@microsoft.com 2barjun@cse.iitm.ac.in Abstract We propose a novel geometric approach for learning bilingual mappings given monolin- gual embeddings and a bilingual dictionary. Our approach decouples the source-to-target language transformation into (a) language- specific rotations on the original embeddings to align them in a common, latent space, and (b) a language-independent similarity met- ric in this common space to better model the similarity between the embeddings. Over- all, we pose the bilingual mapping prob- lem as a classification problem on smooth Riemannian manifolds. Empirically, our ap- proach outperforms previous approaches on the bilingual lexicon induction and cross- lingual word similarity tasks. We next generalize our framework to rep- resent multiple languages in a common latent space. Language-specific rotations for all the languages and a common similarity metric in the latent space are learned jointly from bilingual dictionaries for multiple language pairs. We illustrate the effectiveness of joint learning for multiple languages in an indirect word translation setting. 1 Introduction Bilingual word embeddings are a useful tool in natural language processing (NLP) that has attracted a lot of interest lately due to a fun- damental property: similar concepts/words across ∗This work was carried out during the author’s internship at Microsoft, India. different languages are mapped close to each other in a common embedding space. Hence, they are useful for joint/transfer learning and sharing annotated data across languages different NLP applications such as machinein translation (Gu et al., 2018), building bilingual dictionaries (Mikolov et al., 2013b), mining par- allel corpora (Conneau et al., 2018), text clas- sification (Klementiev et al., 2012), sentiment analysis (Zhou et al., 2015), and dependency parsing (Ammar et al., 2016). Mikolov et al. (2013b) empirically show that a linear transformation of embeddings from one language to another preserves the geometric ar- rangement of word embeddings. In a supervised setting, the transformation matrix, W, is learned given a small bilingual dictionary and their corresponding monolingual embeddings. Subse- quently, many refinements to the bilingual map- ping framework have been proposed (Xing et al., 2015; Smith et al., 2017b; Conneau et al., 2018; Artetxe et al., 2016, 2017, 2018a,b). In this work, we propose a novel geometric ap- proach for learning bilingual embeddings. We ro- tate the source and target language embeddings from their original vector spaces to a common latent space via language-specific orthogonal transformations. Furthermore, we define a sim- ilarity metric, the Mahalanobis metric, in this common space to refine the notion of similarity between a pair of embeddings. We achieve the above by learning the transformation matrix as follows: W = UtBU⊤ s, where Ut and Us are the orthogonal transformations for target and source language embeddings, respectively, and B is a positive definite matrix representing the Mahalanobis metric. 107 Transactions of the Association for Computational Linguistics, vol. 7, pp. 107–120, 2019. Action Editor: Doug Downey. Submission batch: 10/2018; Revision batch: 11/2018; Published 4/2019. c⃝ 2019 …

Content truncated to 500 words