A Latent Variable Model Approach to PMI-based Word Embeddings

A Latent Variable Model Approach to PMI-based Word Embeddings Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski Computer Science Department, Princeton University 35 Olden St, Princeton, NJ 08540 {arora,yuanzhil,yingyul,tengyu,risteski}@cs.princeton.edu Abstract Semantic word embeddings represent the meaning of a word via a vector, and are cre- ated by diverse methods. Many use non- linear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of Mnih and Hinton (2007). The method- ological novelty is to use the prior to com- pute closed form expressions for word statis- tics. This provides a theoretical justifica- tion for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparame- ter choices. It also helps explain why low- dimensional semantic embeddings contain lin- ear algebraic structure that allows solution of word analogies, as shown by Mikolov et al. (2013a) and many subsequent papers. Experimental support is provided for the gen- erative model assumptions, the most impor- tant of which is that latent word vectors are fairly uniformly dispersed in space. 1 Introduction Vector representations of words (word embeddings) try to capture relationships between words as dis- tance or angle, and have many applications in com- putational linguistics and machine learning. They are constructed by various models whose unify- ing philosophy is that the meaning of a word is defined by “the company it keeps” (Firth, 1957), namely, co-occurrence statistics. The simplest meth- ods use word vectors that explicitly represent co- occurrence statistics. Reweighting heuristics are known to improve these methods, as is dimension reduction (Deerwester et al., 1990). Some reweight- ing methods are nonlinear, which include taking the square root of co-occurrence counts (Rohde et al., 2006), or the logarithm, or the related Pointwise Mu- tual Information (PMI) (Church and Hanks, 1990). These are collectively referred to as Vector Space Models, surveyed in (Turney and Pantel, 2010). Neural network language models (Rumelhart et al., 1986; Rumelhart et al., 1988; Bengio et al., 2006; Collobert and Weston, 2008a) propose an- other way to construct embeddings: the word vec- tor is simply the neural network’s internal repre- sentation for the word. This method is nonlinear and nonconvex. It was popularized via word2vec, a family of energy-based models in (Mikolov et al., 2013b; Mikolov et al., 2013c), followed by a ma- trix factorization approach called GloVe (Penning- ton et al., 2014). The first paper also showed how to solve analogies using linear algebra on word em- beddings. Experiments and theory were used to sug- gest that these newer methods are related to the older PMI-based models, but with new hyperparameters and/or term reweighting methods (Levy and Gold- berg, 2014b). But note that even the old PMI method is a bit mysterious. The simplest version considers a sym- metric matrix with each row/column indexed by a word. The entry for (w, w′) is PMI(w, w′) = log p(w,w′) p(w)p(w′), where p(w, w′) is the empirical prob- ability of words w, …

Content truncated to 500 words