Geometry-aware domain adaptation for unsupervised alignment of word embeddings

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3052–3058 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3052 Geometry-aware Domain Adaptation for Unsupervised Alignment of Word Embeddings Pratik Jawanpuria, Mayank Meghwanshi, Bamdev Mishra Microsoft, India {pratik.jawanpuria,mamegh,bamdevm}@microsoft.com Abstract We propose a novel manifold based geometric approach for learning unsupervised alignment of word embeddings between the source and the target languages. Our approach formulates the alignment learning problem as a domain adaptation problem over the manifold of dou- bly stochastic matrices. This viewpoint arises from the aim to align the second order infor- mation of the two language spaces. The rich geometry of the doubly stochastic manifold allows to employ efficient Riemannian conju- gate gradient algorithm for the proposed for- mulation. Empirically, the proposed approach outperforms state-of-the-art optimal transport based approach on the bilingual lexicon induc- tion task across several language pairs. The performance improvement is more significant for distant language pairs. 1 Introduction Learning bilingual word embeddings is an im- portant problem in natural language processing (Mikolov et al., 2013; Faruqui and Dyer, 2014; Artetxe et al., 2016; Conneau et al., 2018), with usage in cross-lingual information retrieval (Vuli´c and Moens, 2015), text classification (Wan et al., 2011; Klementiev et al., 2012), machine transla- tion (Artetxe et al., 2018c) etc. Given a source- target language pair, the aim is to represent the words in both languages in a common embedding space. This is usually achieved by learning a linear function that maps word embeddings of one lan- guage to the embedding space of the other language (Mikolov et al., 2013). Several works have focused on learning such bilingual mapping in supervised setting, using a bilingual dictionary during the training phase (Artetxe et al., 2018a; Joulin et al., 2018; Jawan- puria et al., 2019). Recently, unsupervised bilin- gual word embeddings have also been explored (Zhang et al., 2017a,b; Conneau et al., 2018; Artetxe et al., 2018b; Hoshen and Wolf, 2018; Grave et al., 2019; Alvarez-Melis and Jaakkola, 2018; Zhou et al., 2019; Jawanpuria et al., 2020). Learning unsupervised cross-lingual mapping may be viewed as an instance of the more gen- eral unsupervised domain adaptation problem (Ben- David et al., 2007; Gopalan et al., 2011; Sun et al., 2016; Mahadevan et al., 2018). The latter funda- mentally aims at aligning the input feature (em- beddings) distributions of the source and target domains (languages). In this paper, we take this point of view and learn cross-lingual word align- ment by finding alignment between the second or- der statistics of the source and the target language embedding space. We formulate a novel optimization problem on the set of doubly stochastic matrices. The objec- tive function consists of matching covariances of words from source to target languages in a least- squares sense. For optimization, we exploit the fact that the set of doubly stochastic matrices has rich geometry and forms a Riemannian manifold (Douik and Hassibi, 2019). The Riemannian opti- mization framework (Absil et al., 2008; Edelman et al., …

Content truncated to 500 words