Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 497–507, Dublin, Ireland, August 23-29 2014. Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources Jiang Guo†, Wanxiang Che†, Haifeng Wang‡, Ting Liu†∗ †Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China ‡Baidu Inc., Beijing, China {jguo, car, tliu}@ir.hit.edu.cn wanghaifeng@baidu.com Abstract Recent work has shown success in learning word embeddings with neural network language models (NNLM). However, the majority of previous NNLMs represent each word with a single embedding, which fails to capture polysemy. In this paper, we address this problem by represent- ing words with multiple and sense-specific embeddings, which are learned from bilingual parallel data. We evaluate our embeddings using the word similarity measurement and show that our ap- proach is significantly better in capturing the sense-level word similarities. We further feed our embeddings as features in Chinese named entity recognition and obtain noticeable improvements against single embeddings. 1 Introduction Word embeddings are conventionally defined as compact, real-valued, and low-dimensional vector rep- resentations for words. Each dimension of word embedding represents a latent feature of the word, hope- fully capturing useful syntactic and semantic characteristics. Word embeddings can be used straightfor- wardly for computing word similarities, which benefits many practical applications (Socher et al., 2011; Mikolov et al., 2013a). They are also shown to be effective as input to NLP systems (Collobert et al., 2011) or as features in various NLP tasks (Turian et al., 2010; Yu et al., 2013). In recent years, neural network language models (NNLMs) have become popular architectures for learning word embeddings (Bengio et al., 2003; Mnih and Hinton, 2008; Mikolov et al., 2013b). Most of the previous NNLMs represent each word with a single embedding, which ignores polysemy. In an attempt to better capture the multiple senses or usages of a word, several multi-prototype models have been proposed (Reisinger and Mooney, 2010; Huang et al., 2012). These multi-prototype models simply induce K prototypes (embeddings) for every word in the vocabulary, where K is predefined as a fixed value. These models still may not capture the real senses of words, because different words may have different number of senses. We present a novel and simple method of learning sense-specific word embeddings by using bilingual parallel data. In this method, word sense induction (WSI) is performed prior to the training of NNLMs. We exploit bilingual parallel data for WSI, which is motivated by the intuition that the same word in the source language with different senses is supposed to have different translations in the foreign language.1 For instance, 制服 can be translated as investment / overpower / subdue / subjugate / uniform, etc. Among all of these translations, subdue / overpower / subjugate express the same sense of 制服, whereas uniform / investment express a different sense. Therefore, we could effectively obtain the senses of one word by clustering its translation words, exhibiting different senses in different clusters. The created clusters are then projected back into the words …

Content truncated to 500 words