Context-Dependent Sense Embedding

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 183–191, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Context-Dependent Sense Embedding∗ Lin Qiu† and Kewei Tu‡ and Yong Yu† † Shanghai Jiao Tong University, Shanghai, China, {lqiu,yyu}@apex.sjtu.edu.cn ‡ ShanghaiTech University, Shanghai, China, tukw@shanghaitech.edu.cn Abstract Word embedding has been widely studied and proven helpful in solving many natural lan- guage processing tasks. However, the ambi- guity of natural language is always a prob- lem on learning high quality word embed- dings. A possible solution is sense embed- ding which trains embedding for each sense of words instead of each word. Some re- cent work on sense embedding uses context clustering methods to determine the senses of words, which is heuristic in nature. Other work creates a probabilistic model and per- forms word sense disambiguation and sense embedding iteratively. However, most of the previous work has the problems of learning sense embeddings based on imperfect word embeddings as well as ignoring the depen- dency between sense choices of neighboring words. In this paper, we propose a novel probabilistic model for sense embedding that is not based on problematic word embedding of polysemous words and takes into account the dependency between sense choices. Based on our model, we derive a dynamic program- ming inference algorithm and an Expectation- Maximization style unsupervised learning al- gorithm. The empirical studies show that our model outperforms the state-of-the-art model on a word sense induction task by a 13% rela- tive gain. 1 Introduction Distributed representation of words (aka word em- bedding) aims to learn continuous-valued vectors to ∗The second author was supported by the National Natural Science Foundation of China (61503248). represent words based on their context in a large cor- pus. They can serve as input features for algorithms of natural language processing (NLP) tasks. High quality word embeddings have been proven helpful in many NLP tasks (Collobert and Weston, 2008; Turian et al., 2010; Collobert et al., 2011; Maas et al., 2011; Chen and Manning, 2014). Recently, with the development of deep learning, many novel neural network architectures are proposed for train- ing high quality word embeddings (Mikolov et al., 2013a; Mikolov et al., 2013b). However, since natural language is intrinsically ambiguous, learning one vector for each word may not cover all the senses of the word. In the case of a multi-sense word, the learned vector will be around the average of all the senses of the word in the em- bedding space, and therefore may not be a good rep- resentation of any of the senses. A possible solution is sense embedding which trains a vector for each sense of a word. There are two key steps in training sense embeddings. First, we need to perform word sense disambiguation (WSD) or word sense induc- tion (WSI) to determine the senses of words in the training corpus. Then, we need to train embedding vectors for word senses according to their contexts. Early work on sense embedding (Reisinger and …

Content truncated to 500 words