Joint Learning of Sense and Word Embeddings

Joint Learning of Sense and Word Embeddings Mohammed Alsuhaibani Danushka Bollegala Department of Computer Science, University of Liverpool. {m.a.alsuhaibani, danushka.bollegala}@liverpool.ac.uk Abstract Methods for learning lower-dimensional representations (embeddings) of words using unlabelled data have received a renewed interested due to their myriad success in various Natural Language Processing (NLP) tasks. However, despite their success, a common deficiency associated with most word embedding learning methods is that they learn a single representation for a word, ignoring the different senses of that word (polysemy). To address the polysemy problem, we propose a method that jointly learns sense-aware word embeddings using both unlabelled and sense-tagged text corpora. In particular, our proposed method can learn both word and sense embeddings by efficiently exploiting both types of resources. Our quantitative and qualitative experimental results using unlabelled text corpus with (a) manually annotated word senses, and (b) pseudo annotated senses demonstrate that the proposed method can correctly learn the multiple senses of an ambiguous word. Moreover, the word embeddings learnt by our proposed method outperform several previously proposed competitive word embedding learning methods on word similarity and short-text classification benchmark datasets. Keywords: Sense Embeddings, Word embeddings, Labelled Data, Unlabelled Data 1. Introduction The ability to accurately represent the meanings of words is a fundamental requirement for many natural language processing (NLP) tasks. By using accurate word repre- sentations, it is possible to improve the performance of downstream NLP applications such as name entity recog- nition (NER) (Turian et al., 2010), word similarity mea- surement (Huang et al., 2012), sentiment analysis (Dhillon et al., 2015), word analogy detection (Bollegala et al., 2014), syntactic parsing (Socher et al., 2013) and de- pendency parsing (Bansal et al., 2014). Moreover, com- positional approaches can be used to compute phrase-, sentence- or document-level embeddings from word em- beddings (Baroni et al., 2014). Consequently, various methods have been proposed recently that embed words in lower-dimensional dense vector spaces, for example, using word co-occurrence information such as skip-gram with negative sampling (SGNS), continuous bag-of-words model (CBOW) (Mikolov et al., 2013) and Global Vectors (GloVe) (Pennington et al., 2014), to name a few. A common limitation associated with existing prediction- based word embedding learning methods is that they repre- sent each word by a single vector, ignoring the potentially multiple senses of a word. For example, consider the am- biguous word bank that could mean either a financial in- stitution or a river-bank. The two senses of bank are sig- nificantly different, and embedding both senses to the same point is inadequate. Several solutions have been proposed in the literature to overcome this limitation and learn sense embeddings, which capture the sense related information of words. For example, Reisinger and Mooney (2010) proposed a method for learning sense-specific high dimensional distributional vector representations of words, which was later extended by Huang et al. (2012) using global and local context to learn multiple sense embeddings for an ambiguous word. Neelakantan et al. (2014) proposed multi-sense skip-gram (MSSG), an online cluster-based sense-specific word rep- resentations learning method, …

Content truncated to 500 words