Modeling Word Meaning in Context with Substitute Vectors

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 472–482, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Modeling Word Meaning in Context with Substitute Vectors Oren Melamud Computer Science Dept. Bar-Ilan University melamuo@cs.biu.ac.il Ido Dagan Computer Science Dept. Bar-Ilan University dagan@cs.biu.ac.il Jacob Goldberger Faculty of Engineering Bar-Ilan University goldbej@eng.biu.ac.il Abstract Context representations are a key element in distributional models of word meaning. In contrast to typical representations based on neighboring words, a recently proposed ap- proach suggests to represent a context of a tar- get word by a substitute vector, comprising the potential fillers for the target word slot in that context. In this work we first propose a vari- ant of substitute vectors, which we find partic- ularly suitable for measuring context similar- ity. Then, we propose a novel model for rep- resenting word meaning in context based on this context representation. Our model outper- forms state-of-the-art results on lexical substi- tution tasks in an unsupervised setting. 1 Introduction Following the distributional hypothesis (Firth, 1957), distributional models represent the meaning of a word type as an aggregation of its contexts. A recent line of work addresses polysemy of word types by representing the meaning (or sense) of each word instance individually as induced by its partic- ular context. The context-sensitive meaning of a word instance is commonly called the word meaning in context, as opposed to the word meaning out-of- context of a word type. A key element of distributional models is the choice of context representation. A context of a word instance is typically represented by an unordered collection of its first-order neighboring words, called bag-of-words (BOW). In contrast, Yat- baz et al. (2012) proposed to represent this context as a second-order substitute vector. Instead of the neighboring words themselves, a substitute vector our proposals will unlock new ways of raising the finance needed to develop businesses. representations of context BOW proposals, raising, unlock, ways substitutes alternate, proposing, infinite, various representation of word meaning in context paraphrases new, innovative, different, alternative, novel Table 1: Example for BOW and substitute vector repre- sentations for a context of the target word new. The para- phrase vector is the representation learned by our model for the meaning of new in this context. Only the first few entries for each vector are shown. includes the potential filler words for the target word slot, weighted according to how ‘fit’ they are to fill the target slot given the neighboring words. For ex- ample, the substitute vector representing the con- text “I my smartphone.” (target slot underlined), would typically include potential slot fillers such as love, lost, upgraded, etc. Melamud et al. (2014) argued that substitute vec- tors are potentially more informative than tradi- tional context representations since the fitness of the fillers is estimated using an n-gram language model, thereby capturing information embedded in the neighboring word order. They showed promising results on measuring word similarity out-of-context with a distributional …

Content truncated to 500 words