Embedding Words in Non-Vector Space with Unsupervised Graph Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7317–7331, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7317 Embedding Words in Non-Vector Space with Unsupervised Graph Learning Max Ryabinin1,2 Sergei Popov1,2 Liudmila Prokhorenkova1,2,3 Elena Voita4,5 1Yandex, Russia 2National Research University Higher School of Economics, Russia 3Moscow Institute of Physics and Technology, Russia 4University of Edinburgh, Scotland 5University of Amsterdam, Netherlands {mryab, sapopov, ostroumova-la, lena-voita}@yandex-team.ru Abstract It has become a de-facto standard to represent words as elements of a vector space (word2vec, GloVe). While this approach is convenient, it is unnatural for language: words form a graph with a latent hierarchical structure, and this structure has to be revealed and encoded by word embeddings. We introduce Graph- Glove: unsupervised graph word representa- tions which are learned end-to-end. In our set- ting, each word is a node in a weighted graph and the distance between words is the short- est path distance between the corresponding nodes. We adopt a recent method learning a representation of data in the form of a differen- tiable weighted graph and use it to modify the GloVe training algorithm. We show that our graph-based representations substantially out- perform vector-based methods on word simi- larity and analogy tasks. Our analysis reveals that the structure of the learned graphs is hier- archical and similar to that of WordNet, the ge- ometry is highly non-trivial and contains sub- graphs with different local topology.1 1 Introduction Effective word representations are a key compo- nent of machine learning models for most natu- ral language processing tasks. The most popu- lar approach to represent a word is to map it to a low-dimensional vector (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017; Tifrea et al., 2019). Several algorithms can pro- duce word embedding vectors with distances or dot products capturing semantic relationships be- tween words; the vector representations can be use- ful for solving numerous NLP tasks such as word analogy (Mikolov et al., 2013b), hypernymy detec- 1The training algorithm, preprocessing scripts and evalua- tion benchmarks are available at https://github.com/ yandex-research/graph-glove tion (Tifrea et al., 2019) or serving as features for supervised learning problems. While representing words as vectors may be con- venient, it is unnatural for language: words form a graph with a hierarchical structure (Miller, 1995) that has to be revealed and encoded by unsuper- vised learned word embeddings. A possible step to- wards this can be made by choosing a vector space more similar to the structure of the data: for ex- ample, a space with hyperbolic geometry (Dhingra et al., 2018; Tifrea et al., 2019) instead of com- monly used Euclidean (Mikolov et al., 2013b; Pen- nington et al., 2014; Bojanowski et al., 2017) was shown beneficial for several tasks. However, learn- ing data structure by choosing an appropriate vector space is likely to be neither optimal nor general- izable: Gu et al. (2018) argue that not only are different data better modelled by different spaces, but even for the …

Content truncated to 500 words