Modeling Context Words as Regions: An Ordinal Regression Approach to Word Embedding

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 123–133, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Modeling Context Words as Regions: An Ordinal Regression Approach to Word Embedding Shoaib Jameel and Steven Schockaert School of Computer Science and Informatics Cardiff University {JameelS1, SchockaertS1}@cardiff.ac.uk Abstract Vector representations of word meaning have found many applications in the field of natural language processing. Word vec- tors intuitively represent the average con- text in which a given word tends to oc- cur, but they cannot explicitly model the diversity of these contexts. Although re- gion representations of word meaning of- fer a natural alternative to word vectors, only few methods have been proposed that can effectively learn word regions. In this paper, we propose a new word embedding model which is based on SVM regression. We show that the underlying ranking in- terpretation of word contexts is sufficient to match, and sometimes outperform, the performance of popular methods such as Skip-gram. Furthermore, we show that by using a quadratic kernel, we can effec- tively learn word regions, which outper- form existing unsupervised models for the task of hypernym detection. 1 Introduction Word embedding models such as Skip-gram (Mikolov et al., 2013b) and GloVe (Pennington et al., 2014) represent words as vectors of typi- cally around 300 dimensions. The relatively low- dimensional nature of these word vectors makes them ideally suited for representing textual in- put to neural network models (Goldberg, 2016; Nayak, 2015). Moreover, word embeddings have been found to capture many interesting regulari- ties (Mikolov et al., 2013b; Kim and de Marn- effe, 2013; Gupta et al., 2015; Rothe and Sch¨utze, 2016), which makes it possible to use them as a source of semantic and linguistic knowledge, and to align word embeddings with visual features (Frome et al., 2013) or across different languages (Zou et al., 2013; Faruqui and Dyer, 2014). Notwithstanding the practical advantages of representing words as vectors, a few authors have advocated the idea that words may be better repre- sented as regions (Erk, 2009), possibly with grad- ual boundaries (Vilnis and McCallum, 2015). One important advantage of region representations is that they can distinguish words with a broad mean- ing from those with a more narrow meaning, and should thus in principle be better suited for tasks such as hypernym detection and taxonomy learn- ing. However, it is currently not well understood how such region based representations can best be learned. One possible approach, suggested in (Vil- nis and McCallum, 2015), is to learn a multivari- ate Gaussian for each word, essentially by requir- ing that words which frequently occur together are represented by similar Gaussians. However, for large vocabularies, this is computationally only feasible with diagonal covariance matrices. In this paper, we propose a different approach to learning region representations for words, which is inspired by a geometric view of the Skip-gram model. Essentially, Skip-gram learns two vectors pw and ˜ pw for each word w, such …

Content truncated to 500 words