Enriching Word Sense Embeddings with Translational Context

Proceedings of Recent Advances in Natural Language Processing, pages 208–215, Hissar, Bulgaria, Sep 7–9 2015. Enriching Word Sense Embeddings with Translational Context Mehdi Ghanimifard Department of Philosophy, Linguistics and Theory of Science University of Gothenburg mmehdi.g@gmail.com Richard Johansson Spr˚akbanken, Department of Swedish University of Gothenburg richard.johansson@gu.se Abstract Vector-space models derived from corpora are an effective way to learn a representa- tion of word meaning directly from data, and these models have many uses in prac- tical applications. A number of unsu- pervised approaches have been proposed to automatically learn representations of word senses directly from corpora, but since these methods use no information but the words themselves, they sometimes miss distinctions that could be possible to make if more information were available. In this paper, we present a general frame- work that we call context enrichment that incorporates external information during the training of multi-sense vector-space models. Our approach is agnostic as to which external signal is used to enrich the context, but in this work we consider the use of translations as the source of enrich- ment. We evaluated the models trained us- ing the translation-enriched context using several similarity benchmarks and a word analogy test set. In all our evaluations, the enriched model outperformed the purely word-based baseline soundly. 1 Introduction Word meaning representations derived from cor- pora have recently seen much attention in natural language processing (NLP), most importantly be- cause they can be used very effectively to abstract over the word level in lexicalized NLP systems (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010; Bansal et al., 2014; Guo et al., 2014; Sienˇcnik, 2015). These representations are de- rived from corpus statistics, building on the distri- butional hypothesis that the meaning of a word is reflected in statistical distributions of the contexts in which it appears (Harris, 1954). This intuition can be implemented in a number of ways in prac- tice; in this work, we focus on models that rep- resent word meaning as a point in a metric space (Widdows, 2005; Sahlgren, 2006; Turney and Pan- tel, 2010; Clark, 2015). In particular, one member of this family that has been particularly influen- tial recently is the skip-gram learning algorithm (Mikolov et al., 2013a), which is derived from the log-bilinear language model by Mnih and Hinton (2007). The main reasons for its popularity are its computational efficiency (Mikolov et al., 2013b), its high performance in several evaluations, and the availability of an implementation in the form of the easily usable word2vec package. In most cases distributional word representa- tions disregard the fact that many words have more than one possible interpretation, or word sense, and in lexicographical descriptions of a language we will typically list the senses of a word in dif- ferent sub-entries (Cruse, 1986). For instance, the English word bass can refer to a fish, a musical instrument, the low part of a musical range, etc. It is imaginable that we could use standard tech- niques to learn a vector-space …

Content truncated to 500 words