Towards Understanding Linear Word Analogies

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3253–3262 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3253 Towards Understanding Linear Word Analogies Kawin Ethayarajh, David Duvenaud†, Graeme Hirst University of Toronto †Vector Institute {kawin, duvenaud, gh}@cs.toronto.edu Abstract A surprising property of word vectors is that word analogies can often be solved with vector arithmetic. However, it is unclear why arith- metic operators correspond to non-linear em- bedding models such as skip-gram with neg- ative sampling (SGNS). We provide a formal explanation of this phenomenon without mak- ing the strong assumptions that past theories have made about the vector space and word distribution. Our theory has several implica- tions. Past work has conjectured that linear substructures exist in vector spaces because re- lations can be represented as ratios; we prove that this holds for SGNS. We provide novel justification for the addition of SGNS word vectors by showing that it automatically down- weights the more frequent word, as weighting schemes do ad hoc. Lastly, we offer an in- formation theoretic interpretation of Euclidean distance in vector spaces, justifying its use in capturing word dissimilarity. 1 Introduction Distributed representations of words are a corner- stone of current methods in natural language pro- cessing. Word embeddings, also known as word vectors, can be generated by a variety of models, all of which share Firth’s philosophy (1957) that the meaning of a word is defined by “the company it keeps”. The simplest such models obtain word vectors by constructing a low-rank approximation of a matrix containing a co-occurrence statistic (Landauer and Dumais, 1997; Rohde et al., 2006). In contrast, neural network models (Bengio et al., 2003; Mikolov et al., 2013b) learn word embed- dings by trying to predict words using the contexts they appear in, or vice-versa. A surprising property of word vectors learned via neural networks is that word analogies can of- ten be solved with vector arithmetic. For example, ‘king is to ? as man is to woman’ can be solved by finding the closest vector to ⃗ king− ⃗ man+ ⃗ woman, which should be ⃗ queen. It is unclear why arith- metic operators can effectively compose embed- dings generated by non-linear models such as skip-gram with negative sampling (SGNS). There have been two attempts to rigorously explain this phenomenon, but both have made strong assump- tions about either the embedding space or the word distribution. The paraphrase model (Gittens et al., 2017) hinges on words having a uniform distribu- tion rather than the typical Zipf distribution, which the authors themselves acknowledge is unrealistic. The latent variable model (Arora et al., 2016) as- sumes that word vectors are known a priori and generated by randomly scaling vectors sampled from the unit sphere. In this paper, we explain why – and under what conditions – word analogies can be solved with vector arithmetic, without making the strong as- sumptions past work has. We focus on GloVe and SGNS because they implicitly …

Content truncated to 500 words