On Modeling Sense Relatedness in Multi-prototype Word Embedding

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 233–242, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP On Modeling Sense Relatedness in Multi-prototype Word Embedding Yixin Cao and Juanzi Li∗and Jiaxin Shi and Zhiyuan Liu and Chengjiang Li Dept. of Computer Science and Technology, Tsinghua University, China 100084 {cao-yx13,ljz,shi-jx,liuzy,licj17}@mail.tsinghua.edu.cn Abstract To enhance the expression ability of distributional word representation learn- ing model, many researchers tend to in- duce word senses through clustering, and learn multiple embedding vectors for each word, namely multi-prototype word em- bedding model. However, most related work ignores the relatedness among word senses which actually plays an impor- tant role. In this paper, we propose a novel approach to capture word sense re- latedness in multi-prototype word embed- ding model. Particularly, we differenti- ate the original sense and extended senses of a word by introducing their global oc- currence information and model their re- latedness through the local textual con- text information. Based on the idea of fuzzy clustering, we introduce a ran- dom process to integrate these two types of senses and design two non-parametric methods for word sense induction. To make our model more scalable and ef- ficient, we use an online joint learning framework extended from the Skip-gram model. The experimental results demon- strate that our model outperforms both conventional single-prototype embedding models and other multi-prototype embed- ding models, and achieves more stable performance when trained on smaller data. 1 Introduction Word embedding, representing words in a low di- mentional vector space, plays an increasing im- portant role in various IR and NLP related tasks, such as language modeling (Bengio et al., 2006; ∗Corresponding author. Mnih and Hinton, 2009), named entity recog- nition and disambiguation (Turian et al., 2010; Collobert et al., 2011), and syntactic parsing (Socher et al., 2011, 2013). This trend has been accelerated by the CBOW and the Skip- gram models of (Mikolov et al., 2013b,a) due to its efficiency and remarkable semantic composi- tionality of embedding vectors (e.g. vec(king)- vec(queen)=vec(man)-vec(woman)). However, the assumption that each word is represented by only one single vector is problematic when deal- ing with the polysemous words. Figure 1: Relatedness among senses of the word “book”. To enhance the expression ability of the embed- ding model, recent research has a rising enthusi- asm for representing words at sense level. That is, an individual word is represented as multiple vectors, where each vector corresponds to one of its meanings. Pervious work mostly focus on us- ing clustering to induce word senses (each clus- ter refers to one of the senses) and then learn the word sense representations respectively (Reisinger and Mooney, 2010; Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Li and Jurafsky, 2015). However, the above approaches ignore the relatedness among the word senses. Hence the fol- lowing limitations arise in the usage of hard clus- tering. First of all, many clustering errors will be caused by using hard clustering based method because the …

Content truncated to 500 words