A Probabilistic Model for Learning Multi-Prototype Word Embeddings

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 151–160, Dublin, Ireland, August 23-29 2014. A Probabilistic Model for Learning Multi-Prototype Word Embeddings Fei Tian†, Hanjun Dai∗, Jiang Bian‡, Bin Gao‡, Rui Zhang⋆, Enhong Chen†, Tie-Yan Liu‡ †University of Science and Technology of China, Hefei, P.R.China ∗Fudan University, Shanghai, P.R.China ‡Microsoft Research, Building 2, No. 5 Danling Street, Beijing, P.R.China ⋆Sun Yat-Sen University, Guangzhou, P.R.China †tianfei@mail.ustc.edu.cn, †cheneh@ustc.edu.cn, ∗daihanjun@gmail.com, ‡{jibian, bingao, tyliu}@microsoft.com, ⋆rayz0620@hotmail.com Abstract Distributed word representations have been widely used and proven to be useful in quite a few natural language processing and text mining tasks. Most of existing word embedding models aim at generating only one embedding vector for each individual word, which, however, limits their effectiveness because huge amounts of words are polysemous (such as bank and star). To address this problem, it is necessary to build multi embedding vectors to represent different meanings of a word respectively. Some recent studies attempted to train multi-prototype word embeddings through clustering context window features of the word. However, due to a large number of parameters to train, these methods yield limited scalability and are inefficient to be trained with big data. In this paper, we introduce a much more efficient method for learning multi embedding vectors for polysemous words. In particular, we first propose to model word polysemy from a probabilistic perspective and integrate it with the highly efficient continuous Skip-Gram model. Under this framework, we design an Expectation-Maximization algorithm to learn the word’s multi embedding vectors. With much less parameters to train, our model can achieve comparable or even better results on word-similarity tasks compared with conventional methods. 1 Introduction Distributed word representations usually refer to low dimensional and dense real value vectors (a.k.a. word embeddings) to represent words, which are assumed to convey semantic information contained in words. With the exploding text data on the Web and fast development of deep neural network technolo- gies, distributed word embeddings have been effectively trained and widely used in a lot of text mining tasks (Bengio et al., 2003) (Morin and Bengio, 2005) (Mnih and Hinton, 2007) (Collobert et al., 2011) (Mikolov et al., 2010) (Mikolov et al., 2013b). While word embedding plays an increasingly important role in many tasks, most of word embedding models, which assume one embedding vector for each individual word, suffer from a critical limitation for modeling tremendous polysemous words (e.g. bank, left, doctor). Using the same embedding vec- tor to represent the different meanings (we will call prototype of a word in the rest of the paper) of a polysemous word is somehow unreasonable and sometimes it even hurts the model’s expression ability. To address this problem, some recent efforts, such as (Reisinger and Mooney, 2010) (Huang et al., 2012), have investigated how to obtain multi embedding vectors for the respective different prototypes of a polysemous word. Specifically, these works usually take a two-step approach: they first train single prototype word representations through a multi-layer neural network with the assumption that …

Content truncated to 500 words