Sparse Overcomplete Word Vector Representations

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1491–1500, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Sparse Overcomplete Word Vector Representations Manaal Faruqui Yulia Tsvetkov Dani Yogatama Chris Dyer Noah A. Smith Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, 15213, USA {mfaruqui,ytsvetko,dyogatama,cdyer,nasmith}@cs.cmu.edu Abstract Current distributed representations of words show little resemblance to theo- ries of lexical semantics. The former are dense and uninterpretable, the lat- ter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We pro- pose methods that transform word vec- tors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are dis- covered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they out- perform the original vectors on benchmark tasks. 1 Introduction Distributed representations of words have been shown to benefit NLP tasks like parsing (Lazari- dou et al., 2013; Bansal et al., 2014), named en- tity recognition (Guo et al., 2014), and sentiment analysis (Socher et al., 2013). The attraction of word vectors is that they can be derived directly from raw, unannotated corpora. Intrinsic evalua- tions on various tasks are guiding methods toward discovery of a representation that captures many facts about lexical semantics (Turney, 2001; Tur- ney and Pantel, 2010). Yet word vectors do not look anything like the representations described in most lexical seman- tic theories, which focus on identifying classes of words (Levin, 1993; Baker et al., 1998; Schuler, 2005) and relationships among word meanings (Miller, 1995). Though expensive to construct, conceptualizing word meanings symbolically is important for theoretical understanding and also when we incorporate lexical semantics into com- putational models where interpretability is de- sired. On the surface, discrete theories seem in- commensurate with the distributed approach, a problem now receiving much attention in compu- tational linguistics (Lewis and Steedman, 2013; Kiela and Clark, 2013; Vecchi et al., 2013; Grefen- stette, 2013; Lewis and Steedman, 2014; Paperno et al., 2014). Our contribution to this discussion is a new, principled sparse coding method that transforms any distributed representation of words into sparse vectors, which can then be transformed into binary vectors (§2). Unlike recent approaches of incorpo- rating semantics in distributional word vectors (Yu and Dredze, 2014; Xu et al., 2014; Faruqui et al., 2015), the method does not rely on any external information source. The transformation results in longer, sparser vectors, sometimes called an “over- complete” representation (Olshausen and Field, 1997). Sparse, overcomplete representations have been motivated in other domains as a way to in- crease separability and interpretability, with each instance (here, a word) having a small number of active dimensions (Olshausen and Field, 1997; Lewicki and Sejnowski, 2000), and to increase stability in the presence of noise (Donoho et al., 2006). Our work builds on recent explorations of …

Content truncated to 500 words