Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6479–6489 August 1–6, 2021. ©2021 Association for Computational Linguistics 6479 Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring Aitor Ormazabal1, Mikel Artetxe2, Aitor Soroa1, Gorka Labaka1, Eneko Agirre1 1HiTZ Center, University of the Basque Country (UPV/EHU) 2Facebook AI Research {aitor.ormazabal,a.soroa,gorka.labaka,e.agirre}@ehu.eus artetxe@fb.com Abstract Recent research on cross-lingual word embed- dings has been dominated by unsupervised mapping approaches that align monolingual embeddings. Such methods critically rely on those embeddings having a similar struc- ture, but it was recently shown that the sepa- rate training in different languages causes de- partures from this assumption. In this pa- per, we propose an alternative approach that does not have this limitation, while requiring a weak seed dictionary (e.g., a list of identical words) as the only form of supervision. Rather than aligning two fixed embedding spaces, our method works by fixing the target language embeddings, and learning a new set of embed- dings for the source language that are aligned with them. To that end, we use an exten- sion of skip-gram that leverages translated con- text words as anchor points, and incorporates self-learning and iterative restarts to reduce the dependency on the initial dictionary. Our approach outperforms conventional mapping methods on bilingual lexicon induction, and obtains competitive results in the downstream XNLI task. 1 Introduction Cross-lingual word embeddings (CLWEs) repre- sent words from two or more languages in a shared space, so that semantically similar words in dif- ferent languages are close to each other. Early work focused on jointly learning CLWEs in two languages, relying on a strong cross-lingual su- pervision in the form of parallel corpora (Luong et al., 2015; Gouws et al., 2015) or bilingual dic- tionaries (Gouws and Søgaard, 2015; Duong et al., 2016). However, these approaches were later su- perseded by offline mapping methods, which sepa- rately train word embeddings in different languages and align them in an unsupervised manner through self-learning (Artetxe et al., 2018; Hoshen and Wolf, 2018) or adversarial training (Zhang et al., 2017; Conneau et al., 2018a). Despite the advantage of not requiring any paral- lel resources, mapping methods critically rely on the underlying embeddings having a similar struc- ture, which is known as the isometry assumption. Several authors have observed that this assumption does not generally hold, severely hindering the per- formance of these methods (Søgaard et al., 2018; Nakashole and Flauger, 2018; Patra et al., 2019). In later work, Ormazabal et al. (2019) showed that this issue arises from trying to align separately trained embeddings, as joint learning methods are not sus- ceptible to it. In this paper, we propose an alternative approach that does not have this limitation, but can still work without any parallel resources. The core idea of our method is to fix the target language embed- dings, and learn aligned embeddings for the source language from scratch. This prevents structural mismatches that result from …

Content truncated to 500 words