Representing words as regions in vector space

Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL), pages 57–65, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Representing words as regions in vector space Katrin Erk University of Texas at Austin katrin.erk@mail.utexas.edu Abstract Vector space models of word meaning typi- cally represent the meaning of a word as a vec- tor computed by summing over all its corpus occurrences. Words close to this point in space can be assumed to be similar to it in meaning. But how far around this point does the region of similar meaning extend? In this paper we discuss two models that represent word mean- ing as regions in vector space. Both represen- tations can be computed from traditional point representations in vector space. We find that both models perform at over 95% F-score on a token classification task. 1 Introduction Vector space models of word meaning (Lund and Burgess, 1996; Landauer and Dumais, 1997; Lowe, 2001; Jones and Mewhort, 2007; Sahlgren and Karl- gren, 2005) represent words as points in a high- dimensional semantic space. The dimensions of the space represent the contexts in which each target word has been observed. Distance between vec- tors in semantic space predicts the degree of seman- tic similarity between the corresponding words, as words with similar meaning tend to occur in simi- lar contexts. Because of this property, vector space models have been used successfully both in com- putational linguistics (Manning et al., 2008; Snow et al., 2006; Gorman and Curran, 2006; Sch¨utze, 1998) and in cognitive science (Landauer and Du- mais, 1997; Lowe and McDonald, 2000; McDon- ald and Ramscar, 2001). Given the known problems with defining globally appropriate senses (Kilgarriff, 1997; Hanks, 2000), vector space models are espe- cially interesting for their ability to represent word meaning without relying on dictionary senses. Vector space models typically compute one vec- tor per target word (what we will call word type vec- tors), summing co-occurrence counts over all corpus tokens of the target. If the target word is polyse- mous, the representation will constitute a union over the uses or senses of the word. Such a model does not provide information on the amount of variance in each dimension: Do values on each dimension vary a lot across occurrences of the target? Also, it does not provide information on co-occurrences of feature values in occurrences of the target. To en- code these two types of information, we study richer models of word meaning in vector space beyond sin- gle point representations. Many models of categorization in psychology represent a concept as a region, characterized by feature vectors with dimension weights (Smith et al., 1988; Hampton, 1991; Nosofsky, 1986). Tak- ing our cue from these approaches, we study two models that represent a word as a region in vector space rather than a point. The first model is one that we have recently introduced for representing hy- ponymy in vector space (Erk, 2009). We now test its suitability as a general region model …

Content truncated to 500 words