Constructing High Quality Sense-specific Corpus and Word Embedding via Unsupervised Elimination of Pseudo Multi-sense

Constructing High Quality Sense-Specific Corpus and Word Embedding via Unsupervised Elimination of Pseudo Multi-sense Haoyue Shi1,2, Xihao Wang1,3, Yuqi Sun1,3, Junfeng Hu1,4 1School of Electronics Engineering and Computer Science 2 Department of Machine Intelligence 3 Department of Computer Science and Technology 4 Key Laboratory of Computational Linguistics, Ministry of Education No 5. Yiheyuan Road, Beijing China, 100871 {hyshi, victorwonder, sun yq, hujf}@pku.edu.cn Abstract Multi-sense word embedding is an important extension of neural word embeddings. By leveraging context of each word instance, multi-prototype version of word embeddings were accomplished to represent the multi-senses. Unfortunately, this kind of context based approach inevitably produces multiple senses which should actually be a single one, suffering from the various context of a word. (Shi et al., 2016) used WordNet to evaluate the neighborhood similarity of each sense pair to detect such pseudo multi-senses. In this paper, a novel framework for unsupervised corpus sense tagging is presented, which mainly contains four steps: (a) train multi-sense word embeddings on the given corpus, using existing multi-sense word embedding frameworks; (b) detect pseudo multi-senses in the obtained embeddings, without requirement to any extra language resources; (c) label each word in the corpus with a specific sense tag, with respect to the result of pseudo multi-sense detection; (d) re-train multi-sense word embeddings with the pre-selected sense tags. We evaluate our framework by training word embeddings with the obtained sense specific corpus. On the tasks of word similarity, word analogy as well as sentence understanding, the embeddings trained on sense-specific corpus obtain better results than the basic strategy which is applied in step (a). Keywords: multi-sense word embedding, word sense discovery, pseudo multi-sense 1. Introduction Distributional word representations (Bengio et al., 2003; Collobert and Weston, 2008; Mnih and Hinton, 2009; Mikolov et al., 2013b) embed words into a high dimen- sional space, where the cosine value (or Euclidean distance) between two vectors can somehow represent the similarity of two words. Multi-sense word embedding is an intuitive extension of distributional representations of words. Most of existing works distinguish different senses of words by their contexts (Sch¨utze, 1998; Huang et al., 2012; Pina and Johansson, 2015; Neelakantan et al., 2014; Li and Jurafsky, 2015; Cheng and Kartsaklis, 2015). Be- sides proposing a Chinese Restaurant Process (CRP) model for multi-sense word embeddings, Li and Jurafsky (2015) also discussed how helpful the multi-sense word em- bedding methods are to improve natural language understanding, and proved that they do help on some tasks. However, these methods have a common defect. Exist- ing multi-sense word embedding models generate large amount of pseudo multi-senses (Shi et al., 2016). This phenomenon leads to the vagueness of such distributional word representations. It also has an intuitive explanation. Here shows three appearances of the word “cat(s)” from Wikipedia: • In many countries, cats are believed to have nine lives, but in Italy, Germany, Greece, Brazil and some Spanish-speaking regions, they are said to have seven lives, while in Turkish and Arabic traditions, the num- ber of lives is six. • Female …

Content truncated to 500 words