Topology of Word Embeddings: Singularities Reflect Polysemy

Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), pages 103–113 Barcelona, Spain (Online), December 12–13, 2020 103 Topology of Word Embeddings: Singularities Reflect Polysemy Alexander Jakubowski Heinrich Heine University D¨usseldorf jakubowskialexander @gmail.com Milica Gaˇsi´c Heinrich Heine University D¨usseldorf gasic @hhu.de Marcus Zibrowius Heinrich Heine University D¨usseldorf marcus.zibrowius @cantab.net Abstract The manifold hypothesis suggests that word vectors live on a submanifold within their am- bient vector space. We argue that we should, more accurately, expect them to live on a pinched manifold: a singular quotient of a manifold obtained by identifying some of its points. The identified, singular points corres- pond to polysemous words, i.e. words with multiple meanings. Our point of view sug- gests that monosemous and polysemous words can be distinguished based on the topology of their neighbourhoods. We present two kinds of empirical evidence to support this point of view: (1) We introduce a topological meas- ure of polysemy based on persistent homo- logy that correlates well with the actual num- ber of meanings of a word. (2) We propose a simple, topologically motivated solution to the SemEval-2010 task on Word Sense Induction & Disambiguation that produces competitive results. 1 Introduction Static word embeddings attempt to represent words by vectors in a high-dimensional vector space Rn in such a way that words of similar meaning are represented by (cosine) similar vectors, and vice versa. According to the manifold hypothesis, we should expect these vectors to lie within a lower- dimensional word space W, a subspace of Rn that resembles a manifold. To what extent this hypo- thesis is true in this and other contexts is the subject of ongoing research (Fefferman et al., 2016). In this paper, we argue and demonstrate that for the word space W, polysemy is a principal obstruction to any strict interpretation of the manifold hypothesis. That polysemy presents a serious obstacle to the creation of adequate word vector representations This work is licensed under a Creative Commons Attri- bution 4.0 International Licence. Licence details: http: //creativecommons.org/licenses/by/4.0/. is clear from the outset. Take, for example, a poly- semous word like “mole”. We would want the vectors representing “birthmark” and “counterspy” to be similar to the vector of “mole”, but not similar to each other. This is impossible. In order for sim- ilarity of vectors to accurately encode similarity in meaning, we need vectors representing meanings, not words. Let us therefore hypothesize a space of mean- ings M that accurately represents all possible meanings and their similarities. Our argument is a simple topological observation based on the rela- tionship between this space M and the word space W. For an idealised language, where there is a bijection between meanings and words, these two spaces would agree. For a natural language, how- ever, multiple points of M get identified with a single point of W. This process corresponds to a to- pological construction that we refer to as pinching (see Figure 2). It is easy to see that a space resulting from …

Content truncated to 500 words