Are All Good Word Vector Spaces Isomorphic?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3178–3192, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3178 Are All Good Word Vector Spaces Isomorphic? Ivan Vuli´c1∗ Sebastian Ruder2∗ Anders Søgaard3,4∗ 1 Language Technology Lab, University of Cambridge 2 DeepMind 3 Department of Computer Science, University of Copenhagen 4 Google Research, Berlin iv250@cam.ac.uk ruder@google.com soegaard@di.ku.dk Abstract Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic. As a result, they perform poorly or fail completely on non- isomorphic spaces. Such non-isomorphism has been hypothesised to result from typolog- ical differences between languages. In this work, we ask whether non-isomorphism is also crucially a sign of degenerate word vector spaces. We present a series of experiments across diverse languages which show that vari- ance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the mono- lingual resources available, and to the proper- ties and duration of monolingual training (e.g. “under-training”). 1 Introduction Word embeddings have been argued to reflect how language users organise concepts (Mandera et al., 2017; Torabi Asr et al., 2018). The extent to which they really do so has been evaluated, e.g., using semantic word similarity and association norms (Hill et al., 2015; Gerz et al., 2016), and word analogy benchmarks (Mikolov et al., 2013c). If word embeddings reflect more or less language- independent conceptual organisations, word em- beddings in different languages can be expected to be near-isomorphic. Researchers have exploited this to learn linear transformations between such spaces (Mikolov et al., 2013a; Glavaˇs et al., 2019), which have been used to induce bilingual dictionar- ies, as well as to facilitate multilingual modeling and cross-lingual transfer (Ruder et al., 2019). In this paper, we show that near-isomorphism arises only with sufficient amounts of training. This is of practical interest for applications of linear ∗All authors contributed equally to this work. 10M 20M 50M 100M 200M 500M # of tokens in Wikipedia 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BLI scores (Accuracy) th lv et ms hi lt sl el sk hr id fa da ko tr ro ar fi he cs vi hu uk sv ca pt pl zh ja nl ru it es fr de Figure 1: Performance of a state-of-the-art BLI model mapping from English to a target language and the size of the target language Wikipedia are correlated. Linear fit shown as a blue line (log scale). alignment methods for cross-lingual word embed- dings. It furthermore provides us with an expla- nation for reported failures to align word vector spaces in different languages (Søgaard et al., 2018; Artetxe et al., 2018a), which has so far been largely attributed only to inherent typological differences. In fact, the amount of data used to induce the monolingual embeddings is predictive of the qual- ity of the aligned cross-lingual word embeddings, as evaluated on bilingual lexicon induction (BLI). Consider, for motivation, Figure 1; it shows the per- …

Content truncated to 500 words