A Simple and Efficient Method to Generate Word Sense Representations

Proceedings of Recent Advances in Natural Language Processing, pages 465–472, Hissar, Bulgaria, Sep 7–9 2015. A Simple and Efficient Method to Generate Word Sense Representations Luis Nieto Pi˜na and Richard Johansson Spr˚akbanken, Department of Swedish, University of Gothenburg Box 200, SE-40530 Gothenburg, Sweden {luis.nieto.pina, richard.johansson}@svenska.gu.se Abstract Distributed representations of words have boosted the performance of many Natu- ral Language Processing tasks. However, usually only one representation per word is obtained, not acknowledging the fact that some words have multiple meanings. This has a negative effect on the individ- ual word representations and the language model as a whole. In this paper we present a simple model that enables recent tech- niques for building word vectors to rep- resent distinct senses of polysemic words. In our assessment of this model we show that it is able to effectively discriminate between words’ senses and to do so in a computationally efficient manner. 1 Introduction Distributed representations of words have helped obtain better language models (Bengio et al., 2003) and improve the performance of many natural language processing applications such as named entity recognition, chunking, paraphrasing, or sentiment classification (Turian et al., 2010; Socher et al., 2011; Glorot et al., 2011). Recently, the Skip-gram model (Mikolov et al., 2013a; Mikolov et al., 2013b) was proposed, which is able to produce high-quality representations from large collections of text in an efficient manner. Despite the achievements of distributed repre- sentations, polysemy or homonymy are usually disregarded even when word semantics may have a large influence on the models. This results in several distinct senses of one same word sharing a representation, and possibly influencing the repre- sentations of words related to those distinct senses under the premise that similar words should have similar representations. Some recent attempts to address this issue are mentioned in the next sec- tion. We present a simple method for obtaining sense representations directly during the Skip- gram training phase. It differs from most previ- ous approaches in that it does not need to create or maintain clusters to discriminate between senses, leading to a significant reduction in the model’s complexity. It also uses a heuristic approach to determining the number of senses to be learned per word that allows the model to use knowledge from lexical resources but also to keep its ability to work withouth them. In the following sections we look at previous work, describe our model, and in- spect its results in qualitative and quantitative eval- uations. 2 Related Work One of the first steps towards obtaining word sense embeddings was that by Reisinger and Mooney (2010). The authors propose to cluster occur- rences of any given word in a corpus into a fixed number K of clusters which represent different word usages (rather than word senses). Each word’s is thus assigned multiple prototypes or em- beddings. Huang et al. (2012) introduced a neural lan- guage model that leverages sentence-level and document-level context to generate word embed- dings. Using Reisinger and Mooney (2010)’s ap- proach to generate …

Content truncated to 500 words