Word2Sense: Sparse Interpretable Word Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5692–5705 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5692 Word2Sense : Sparse Interpretable Word Embeddings Abhishek Panigrahi Microsoft Research India t-abpani@microsoft.com Harsha Vardhan Simhadri Microsoft Research India harshasi@microsoft.com Chiranjib Bhattacharyya Microsoft Research India, and Indian Institute of Science chiru@iisc.ac.in Abstract We present an unsupervised method to gener- ate Word2Sense word embeddings that are interpretable — each dimension of the em- bedding space corresponds to a fine-grained sense, and the non-negative value of the em- bedding along the j-th dimension represents the relevance of the j-th sense to the word. The underlying LDA-based generative model can be extended to refine the representation of a polysemous word in a short context, allowing us to use the embeddings in con- textual tasks. On computational NLP tasks, Word2Sense embeddings compare well with other word embeddings generated by unsuper- vised methods. Across tasks such as word sim- ilarity, entailment, sense induction, and con- textual interpretation, Word2Sense is compet- itive with the state-of-the-art method for that task. Word2Sense embeddings are at least as sparse and fast to compute as prior art. 1 Introduction Several unsupervised methods such as SkipGram (Mikolov et al., 2013) and Glove (Pennington et al., 2014) have demon- strated that co-occurrence data from large corpora can be used to compute low-dimensional repre- sentations of words (a.k.a. embeddings) that are useful in computational NLP tasks. While not as accurate as semi-supervised methods such as BERT (Devlin et al., 2018) and ELMO (Peters et al., 2018) that are trained on various down- stream tasks, they do not require massive amounts of compute unaccessible to all but few. Nearly all such methods produce dense repre- sentations for words whose coordinates in them- selves have no meaningful interpretation. The nu- merical values of a word’s embedding are mean- ingful only in relation to representations of other words. A unitary rotation can be applied to many of these embeddings retaining their utility for computational tasks, and yet completely chang- ing the values of individual coordinates. Can we design an interpretable embedding whose coordi- nates have a clear meaning to humans? Ideally such an embedding would capture the multiple senses of a word, while being effective at computational tasks that use inter-word spacing of embeddings. Loosely, a sense is a set of semanti- cally similar words that collectively evoke a bigger picture than individual words in the reader’s mind. In this work, we mathematically define a sense to be a probability distribution over the vocabulary, just as topics in topic models. A human can re- late to a sense through the words with maximum probability in the sense’s probability distribution. Table 1 presents the top 10 words for a few senses. We describe precisely such an embedding of words in a space where each dimension corre- sponds to a sense. Words are represented as prob- ability distributions over senses so that the magni- tude of each coordinate represents the relative …

Content truncated to 500 words