Analyzing the Limitations of Cross-lingual Word Embedding Mappings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4990–4995 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4990 Analyzing the Limitations of Cross-lingual Word Embedding Mappings Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, Eneko Agirre IXA NLP Group University of the Basque Country (UPV/EHU) aormazabal024@ikasle.ehu.eus {mikel.artetxe, gorka.labaka, a.soroa, e.agirre}@ehu.eus Abstract Recent research in cross-lingual word embed- dings has almost exclusively focused on of- fline methods, which independently train word embeddings in different languages and map them to a shared space through linear trans- formations. While several authors have ques- tioned the underlying isomorphism assump- tion, which states that word embeddings in dif- ferent languages have approximately the same structure, it is not clear whether this is an in- herent limitation of mapping approaches or a more general issue when learning cross- lingual embeddings. So as to answer this question, we experiment with parallel corpora, which allows us to compare offline mapping to an extension of skip-gram that jointly learns both embedding spaces. We observe that, un- der these ideal conditions, joint learning yields to more isomorphic embeddings, is less sen- sitive to hubness, and obtains stronger results in bilingual lexicon induction. We thus con- clude that current mapping methods do have strong limitations, calling for further research to jointly learn cross-lingual embeddings with a weaker cross-lingual signal. 1 Introduction Cross-lingual word embeddings have attracted a lot of attention in recent times. Existing meth- ods can be broadly classified into two categories: joint methods, which simultaneously learn word representations for multiple languages on parallel corpora (Gouws et al., 2015; Luong et al., 2015), and mapping methods, which independently train word embeddings in different languages and map them to a shared space through linear transforma- tions (Mikolov et al., 2013a; Artetxe et al., 2018a). While early work in cross-lingual word embed- dings was dominated by joint approaches, recent research has almost exclusively focused on map- ping methods, which have the advantage of requir- ing little or no cross-lingual signal (Zhang et al., 2017; Conneau et al., 2018; Artetxe et al., 2018b). For mapping methods to work, it is necessary that embedding spaces in different languages have a similar structure (i.e. are approximately isomor- phic), as it would otherwise be hopeless to find a linear map from one space to another. Several authors have questioned this assumption, show- ing that linguistic and domain divergences cause strong mismatches in embedding spaces, which in turn heavily hinders the performance of these methods (Søgaard et al., 2018; Patra et al., 2019). Nevertheless, it is not clear whether this mismatch is a consequence of separately training both em- bedding spaces, and thus an inherent limitation of mapping approaches, or an insurmountable ob- stacle that arises from the linguistic divergences across languages, and hence a more general issue when learning cross-lingual word embeddings. The goal of this paper is to shed light on this matter so as to better understand the nature and extension of these limitations. …

Content truncated to 500 words