A Mixture Model for Learning Multi-Sense Word Embeddings

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 121–127, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics A Mixture Model for Learning Multi-Sense Word Embeddings Dai Quoc Nguyen1, Dat Quoc Nguyen2, Ashutosh Modi1, Stefan Thater1, Manfred Pinkal1 1Department of Computational Linguistics, Saarland University, Germany {daiquocn, ashutosh, stth, pinkal}@coli.uni-saarland.de 2Department of Computing, Macquarie University, Australia dat.nguyen@students.mq.edu.au Abstract Word embeddings are now a standard technique for inducing meaning represen- tations for words. For getting good repre- sentations, it is important to take into ac- count different senses of a word. In this paper, we propose a mixture model for learning multi-sense word embeddings. Our model generalizes the previous works in that it allows to induce different weights of different senses of a word. The experi- mental results show that our model outper- forms previous models on standard evalu- ation tasks. 1 Introduction Word embeddings have shown to be useful in var- ious NLP tasks such as sentiment analysis, topic models, script learning, machine translation, se- quence labeling and parsing (Socher et al., 2013; Sutskever et al., 2014; Modi and Titov, 2014; Nguyen et al., 2015a,b; Modi, 2016; Ma and Hovy, 2016; Nguyen et al., 2017; Modi et al., 2017). A word embedding captures the syntac- tic and semantic properties of a word by repre- senting the word in a form of a real-valued vector (Mikolov et al., 2013a,b; Pennington et al., 2014; Levy and Goldberg, 2014). However, usually word embedding models do not take into account lexical ambiguity. For ex- ample, the word bank is usually represented by a single vector representation for all senses in- cluding sloping land and financial institution. Re- cently, approaches have been proposed to learn multi-sense word embeddings, where each sense of a word corresponds to a sense-specific em- bedding. Reisinger and Mooney (2010), Huang et al. (2012) and Wu and Giles (2015) proposed methods to cluster the contexts of each word and then using cluster centroids as vector representa- tions for word senses. Neelakantan et al. (2014), Tian et al. (2014), Li and Jurafsky (2015) and Chen et al. (2015) extended Word2Vec models (Mikolov et al., 2013a,b) to learn a vector repre- sentation for each sense of a word. Chen et al. (2014), Iacobacci et al. (2015) and Flekova and Gurevych (2016) performed word sense induction using external resources (e.g., WordNet, Babel- Net) and then learned sense embeddings using the Word2Vec models. Rothe and Sch¨utze (2015) and Pilehvar and Collier (2016) presented methods us- ing pre-trained word embeddings to learn embed- dings from WordNet synsets. Cheng et al. (2015), Liu et al. (2015b), Liu et al. (2015a) and Zhang and Zhong (2016) directly opt the Word2Vec Skip- gram model (Mikolov et al., 2013b) for learning the embeddings of words and topics on a topic- assigned corpus. One issue in these previous works is that they assign the same weight to every sense of a word. The central assumption of our work is that each sense of a word …

Content truncated to 500 words