The strange geometry of skip-gram with negative sampling

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2873–2878 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics The strange geometry of skip-gram with negative sampling David Mimno and Laure Thompson Cornell University mimno@cornell.edu, laurejt@cs.cornell.edu Abstract Despite their ubiquity, word embeddings trained with skip-gram negative sampling (SGNS) remain poorly understood. We find that vector positions are not simply deter- mined by semantic similarity, but rather oc- cupy a narrow cone, diametrically opposed to the context vectors. We show that this ge- ometric concentration depends on the ratio of positive to negative examples, and that it is neither theoretically nor empirically inherent in related embedding algorithms. 1 Introduction It is generally assumed that the geometry of word embeddings is determined by semantic relatedness. Vectors are assumed to be distributed throughout a K-dimensional space, with specific regions de- voted to specific concepts. We find that vectors trained with the skip-gram with negative sampling (SGNS) algorithm (Mikolov et al., 2013) are not only influenced by semantics but are also strongly influenced by the negative sampling objective. In fact, far from spanning the possible space, they exist only in a narrow cone in RK. Nevertheless, SGNS vectors have become a foundational tool in NLP and perform as well or better than numerous methods with similar objectives (Turian et al., 2010; Dhillon et al., 2012; Pennington et al., 2014; Luo et al., 2015) with respect to evaluations of intrinsic and extrinsic quality (Schnabel et al., 2015). SGNS works by training two sets of embeddings: the “official” word embeddings and a second set of context embeddings, with one K-dimensional vec- tor in each set for each word in the vocabulary. The objective tries to make the word vector and con- text vector closer for a pair of words that actually occur together than for randomly sampled “nega- tive” words. Following training, the word vectors are typically saved; the context vectors are deleted. Any difference between these two sets of vectors is puzzling, since the sliding window used in training is symmetrical: a word and its context word reverse roles almost immediately. Indeed, the superficially similar GloVe algorithm (Pennington et al., 2014) also defines word and context vectors and by de- fault returns the mean of these two vectors. Previous work has analyzed what the algorithm might be doing in theory, as an approximation to a matrix factorization (Levy and Goldberg, 2014). Other work has considered the empirical effects of some of the more arbitrary-seeming algorithmic choices (Levy et al., 2015). But we still have rel- atively little understanding of how the algorithm actually determines parameter values. as a of used the and is in to any that italso by from are be there this most notor place they however an especially with was on were resources found its individual one six europe sometimes his all at during some other than seven zero three where he which no being nine have known eight four for their such limited s would do …

Content truncated to 500 words