GloVe: Global Vectors for Word Representation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics GloVe: Global Vectors for Word Representation Jeffrey Pennington, Richard Socher, Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA 94305 jpennin@stanford.edu, richard@socher.org, manning@stanford.edu Abstract Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arith- metic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log- bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co- occurrence matrix, rather than on the en- tire sparse matrix or on individual context windows in a large corpus. The model pro- duces a vector space with meaningful sub- structure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on simi- larity tasks and named entity recognition. 1 Introduction Semantic vector space models of language repre- sent each word with a real-valued vector. These vectors can be used as features in a variety of ap- plications, such as information retrieval (Manning et al., 2008), document classification (Sebastiani, 2002), question answering (Tellex et al., 2003), named entity recognition (Turian et al., 2010), and parsing (Socher et al., 2013). Most word vector methods rely on the distance or angle between pairs of word vectors as the pri- mary method for evaluating the intrinsic quality of such a set of word representations. Recently, Mikolov et al. (2013c) introduced a new evalua- tion scheme based on word analogies that probes the finer structure of the word vector space by ex- amining not the scalar distance between word vec- tors, but rather their various dimensions of dif- ference. For example, the analogy “king is to queen as man is to woman” should be encoded in the vector space by the vector equation king − queen = man − woman. This evaluation scheme favors models that produce dimensions of mean- ing, thereby capturing the multi-clustering idea of distributed representations (Bengio, 2009). The two main model families for learning word vectors are: 1) global matrix factorization meth- ods, such as latent semantic analysis (LSA) (Deer- wester et al., 1990) and 2) local context window methods, such as the skip-gram model of Mikolov et al. (2013c). Currently, both families suffer sig- nificant drawbacks. While methods like LSA ef- ficiently leverage statistical information, they do relatively poorly on the word analogy task, indi- cating a sub-optimal vector space structure. Meth- ods like skip-gram may do better on the analogy task, but they poorly utilize the statistics of the cor- pus since they train on separate local context win- dows instead of on global co-occurrence …

Content truncated to 500 words