Linguistic Regularities in Continuous Space Word Representations

Proceedings of NAACL-HLT 2013, pages 746–751, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Linguistic Regularities in Continuous Space Word Representations Tomas Mikolov∗, Wen-tau Yih, Geoffrey Zweig Microsoft Research Redmond, WA 98052 Abstract Continuous space language models have re- cently demonstrated outstanding results across a variety of tasks. In this paper, we ex- amine the vector-space word representations that are implicitly learned by the input-layer weights. We find that these representations are surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset. This allows vector-oriented reasoning based on the offsets between words. For example, the male/female relationship is automatically learned, and with the induced vector representations, “King - Man + Woman” results in a vector very close to “Queen.” We demonstrate that the word vectors capture syntactic regularities by means of syntactic analogy questions (provided with this paper), and are able to correctly answer almost 40% of the questions. We demonstrate that the word vectors capture semantic regu- larities by using the vector offset method to answer SemEval-2012 Task 2 questions. Re- markably, this method outperforms the best previous systems. 1 Introduction A defining feature of neural network language mod- els is their representation of words as high dimen- sional real valued vectors. In these models (Ben- gio et al., 2003; Schwenk, 2007; Mikolov et al., 2010), words are converted via a learned lookup- table into real valued vectors which are used as the ∗Currently at Google, Inc. inputs to a neural network. As pointed out by the original proposers, one of the main advantages of these models is that the distributed representation achieves a level of generalization that is not possi- ble with classical n-gram language models; whereas a n-gram model works in terms of discrete units that have no inherent relationship to one another, a con- tinuous space model works in terms of word vectors where similar words are likely to have similar vec- tors. Thus, when the model parameters are adjusted in response to a particular word or word-sequence, the improvements will carry over to occurrences of similar words and sequences. By training a neural network language model, one obtains not just the model itself, but also the learned word representations, which may be used for other, potentially unrelated, tasks. This has been used to good effect, for example in (Collobert and Weston, 2008; Turian et al., 2010) where induced word rep- resentations are used with sophisticated classifiers to improve performance in many NLP tasks. In this work, we find that the learned word repre- sentations in fact capture meaningful syntactic and semantic regularities in a very simple way. Specif- ically, the regularities are observed as constant vec- tor offsets between pairs of words sharing a par- ticular relationship. For example, if we denote the vector for word i as xi, and focus on the singu- lar/plural relation, we observe that xapple−xapples ≈ xcar−xcars, xfamily−xfamilies ≈ xcar−xcars, and so on. Perhaps more surprisingly, we …

Content truncated to 500 words