Non-Linear Instance-Based Cross-Lingual Mapping for Non-Isomorphic Embedding Spaces

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7548–7555 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7548 Non-Linear Instance-Based Cross-Lingual Mapping for Non-Isomorphic Embedding Spaces Goran Glavaˇs1 and Ivan Vuli´c2 1 Data and Web Science Group, University of Mannheim, Germany 2Language Technology Lab, TAL, University of Cambridge, UK goran@informatik.uni-mannheim.de iv250@cam.ac.uk Abstract We present INSTAMAP, an instance-based method for learning projection-based cross- lingual word embeddings. Unlike prior work, it deviates from learning a single global lin- ear projection. INSTAMAP is a non-parametric model that learns a non-linear projection by iteratively: (1) finding a globally optimal ro- tation of the source embedding space relying on the Kabsch algorithm, and then (2) moving each point along an instance-specific transla- tion vector estimated from the translation vec- tors of the point’s nearest neighbours in the training dictionary. We report performance gains with INSTAMAP over four representa- tive state-of-the-art projection-based models on bilingual lexicon induction across a set of 28 diverse language pairs. We note promi- nent improvements, especially for more dis- tant language pairs (i.e., languages with non- isomorphic monolingual spaces). 1 Introduction and Motivation Induction of cross-lingual word embeddings (CLWEs) (Vuli´c et al., 2011; Mikolov et al., 2013; Xing et al., 2015; Smith et al., 2017; Artetxe et al., 2018) has been one of the key mechanisms for enabling multilingual modeling of meaning and facilitating cross-lingual transfer for downstream NLP tasks. Even though CLWEs are recently be- ing contested in cross-lingual downstream transfer by pretrained multilingual language models (Pires et al., 2019; Conneau et al., 2020; Artetxe et al., 2019; Wu and Dredze, 2019; Wu et al., 2020), they are still paramount in word-level translation, that is, bilingual lexicon induction (BLI). While earlier work focused on joint induction of multilingual embeddings from multilingual cor- pora, relying on word- (Klementiev et al., 2012; Koˇcisk`y et al., 2014; Gouws and Søgaard, 2015), sentence- (Zou et al., 2013; Hermann and Blunsom, 2014; Luong et al., 2015; Coulmance et al., 2015; Levy et al., 2017), or document-level (Søgaard et al., 2015; Mogadala and Rettinger, 2016; Vuli´c and Moens, 2016) alignments, most recent ef- forts focus on post-hoc alignment of independently trained monolingual embeddings (the so-called pro- jection or mapping approaches) (Smith et al., 2017; Artetxe et al., 2018; Conneau et al., 2018; Joulin et al., 2018; Patra et al., 2019, inter alia). Despite some recent evidence that joint CLWE induction may lead to better bilingual spaces (Or- mazabal et al., 2019), projection-based methods still dominate the field (Hoshen and Wolf, 2018; Ruder et al., 2018; Nakashole, 2018; Grave et al., 2019; Zhang et al., 2019, inter alia) due to their conceptual attractiveness: they operate on top of vectors produced with any embedding model and need at most a few thousand word pairs of supervi- sion (Glavaˇs et al., 2019; Vuli´c et al., 2019). Most projection-based CLWE models induce bilingual spaces by orthogonally projecting one monolingual space to another. Since orthogonal projections do not affect the topology of the source …

Content truncated to 500 words