Linguistic Regularities in Sparse and Explicit Word Representations

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 171–180, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Linguistic Regularities in Sparse and Explicit Word Representations Omer Levy∗and Yoav Goldberg Computer Science Department Bar-Ilan University Ramat-Gan, Israel {omerlevy,yoav.goldberg}@gmail.com Abstract Recent work has shown that neural- embedded word representations capture many relational similarities, which can be recovered by means of vector arithmetic in the embedded space. We show that Mikolov et al.’s method of first adding and subtracting word vectors, and then searching for a word similar to the re- sult, is equivalent to searching for a word that maximizes a linear combination of three pairwise word similarities. Based on this observation, we suggest an improved method of recovering relational similar- ities, improving the state-of-the-art re- sults on two recent word-analogy datasets. Moreover, we demonstrate that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word repre- sentations. 1 Introduction Deep learning methods for language processing owe much of their success to neural network lan- guage models, in which words are represented as dense real-valued vectors in Rd. Such representa- tions are referred to as distributed word represen- tations or word embeddings, as they embed an en- tire vocabulary into a relatively low-dimensional linear space, whose dimensions are latent contin- uous features. The embedded word vectors are trained over large collections of text using vari- ants of neural networks (Bengio et al., 2003; Col- lobert and Weston, 2008; Mnih and Hinton, 2008; Mikolov et al., 2011; Mikolov et al., 2013b). The ∗ Supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agree- ment no. 287923 (EXCITEMENT). word embeddings are designed to capture what Turney (2006) calls attributional similarities be- tween vocabulary items: words that appear in sim- ilar contexts will be close to each other in the projected space. The effect is grouping of words that share semantic (“dog cat cow”, “eat devour”) or syntactic (“cars hats days”, “emptied carried danced”) properties, and are shown to be effective as features for various NLP tasks (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011; Al-Rfou et al., 2013). We refer to such word rep- resentations as neural embeddings or just embed- dings. Recently, Mikolov et al. (2013c) demonstrated that the embeddings created by a recursive neu- ral network (RNN) encode not only attributional similarities between words, but also similarities between pairs of words. Such similarities are referred to as linguistic regularities by Mikolov et al. and as relational similarities by Turney (2006). They capture, for example, the gen- der relation exhibited by the pairs “man:woman”, “king:queen”, the language-spoken-in relation in “france:french”, “mexico:spanish” and the past- tense relation in “capture:captured”, “go:went”. Remarkably, Mikolov et al. showed that such rela- tions are reflected in vector offsets between word pairs (apples − apple ≈ cars − car), and that by using simple vector arithmetic one could apply the relation and solve analogy …

Content truncated to 500 words