Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Proceedings of the 19th Conference on Computational Language Learning, pages 258–267, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz,1 Roi Reichart,2 1Institute of Computer Science, The Hebrew University 2Technion, IIT {roys02|arir}@cs.huji.ac.il roiri@ie.technion.ac.il Ari Rappoport1 Abstract We present a novel word level vector rep- resentation based on symmetric patterns (SPs). For this aim we automatically ac- quire SPs (e.g., “X and Y”) from a large corpus of plain text, and generate vectors where each coordinate represents the co- occurrence in SPs of the represented word with another word of the vocabulary. Our representation has three advantages over existing alternatives: First, being based on symmetric word relationships, it is highly suitable for word similarity prediction. Particularly, on the SimLex999 word simi- larity dataset, our model achieves a Spear- man’s ρ score of 0.517, compared to 0.462 of the state-of-the-art word2vec model. In- terestingly, our model performs exception- ally well on verbs, outperforming state- of-the-art baselines by 20.2–41.5%. Sec- ond, pattern features can be adapted to the needs of a target NLP application. For ex- ample, we show that we can easily control whether the embeddings derived from SPs deem antonym pairs (e.g. (big,small)) as similar or dissimilar, an important distinc- tion for tasks such as word classification and sentiment analysis. Finally, we show that a simple combination of the word sim- ilarity scores generated by our method and by word2vec results in a superior predic- tive power over that of each individual model, scoring as high as 0.563 in Spear- man’s ρ on SimLex999. This emphasizes the differences between the signals cap- tured by each of the models. 1 Introduction In the last decade, vector space modeling (VSM) for word representation (a.k.a word embedding), has become a key tool in NLP. Most approaches to word representation follow the distributional hy- pothesis (Harris, 1954), which states that words that co-occur in similar contexts are likely to have similar meanings. VSMs differ in the way they exploit word co- occurrence statistics. Earlier works (see (Turney et al., 2010)) encode this information directly in the features of the word vector representation. More Recently, Neural Networks have become promi- nent in word representation learning (Bengio et al., 2003; Collobert and Weston, 2008; Collobert et al., 2011; Mikolov et al., 2013a; Pennington et al., 2014, inter alia). Most of these models aim to learn word vectors that maximize a language model objective, thus capturing the tendencies of the represented words to co-occur in the training corpus. VSM approaches have resulted in highly useful word embeddings, obtaining high quality results on various semantic tasks (Baroni et al., 2014). Interestingly, the impressive results of these models are achieved despite the shallow linguis- tic information most of them consider, which is limited to the tendency of words to co-occur to- gether in a pre-specified context window. Particu- larly, very little information is encoded about the syntactic and semantic relations between the par- ticipating words, and, instead, …

Content truncated to 500 words