Best of Both Worlds: Making Word Sense Embeddings Interpretable

Best of Both Worlds: Making Word Sense Embeddings Interpretable Alexander Panchenko Language Technology Group, Technische Universit¨at Darmstadt Hochschulstr. 10, 64289, Darmstadt, Germany panchenko@lt.informatik.tu-darmstadt.de Abstract Word sense embeddings represent a word sense as a low-dimensional numeric vector. While this representation is potentially useful for NLP applications, its interpretability is inherently limited. We propose a simple technique that improves interpretability of sense vectors by mapping them to synsets of a lexical resource. Our experiments with AdaGram sense embeddings and BabelNet synsets show that it is possible to retrieve synsets that correspond to automatically learned sense vectors with Precision of 0.87, Recall of 0.42 and AUC of 0.78. Keywords: word sense embeddings, WordNet, BabelNet, AdaGram, sense matching, lexical semantics 1. Introduction Two main approaches to represent the meaning of lexical units, such as words and multiword expressions are lexico- graphy and statistical corpus analysis. In the first approach, a human explicitly encodes lexical-semantic knowledge, usually in the form of synsets (i.e. sets of synonyms), typed relations between synsets and sense definitions. A prominent example of this approach is the Princeton Word- Net (Miller, 1995). The second approach makes use of text corpora to extract relations between words and feature rep- resentations of words and senses. These methods are trying to avoid manual work as much as possible. Whereas lexi- cal resources are manually created, in the second approach most methods extract the information from text without hu- man intervention. Examples of the second group of meth- ods include ”classical” vector-based (Baroni and Lenci, 2010) and symbolic (Biemann and Riedl, 2013) distribu- tional models, as well as word embeddings (Mikolov et al., 2013; Pennington et al., 2014). One of the strongest sides of lexical-semantic resources is their interpretability – they are entirely human-readable and drawn distinctions are motivated by lexicographic or psychological considerations. On the downside, these WordNet-like resources are expensive to create, and it is not easy to adapt them to a given domain of interest or lan- guage. Besides, sense inventories of lexical resources are often too fine grained to be useful in downstream applica- tions (Brown, 2008). At the same time, corpus-driven approaches are strong at adaptivity – they can be re-trained on a new corpus, thus naturally adapting to the domain at hand. If fitted with a word sense induction algorithm, corpus-driven approaches can also discover new senses (Erk et al., 2009). However, the representations they deliver are often not matching the standards of lexicography, and they rather distinguish word usages than senses. Moreover, dense numeric vector repre- sentations as present in latent vector spaces (Sch¨utze, 1998) and word embeddings are barely interpretable. Word sense embeddings (Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014) extend word embeddings so that a word is represented by several vectors corresponding to meanings of the word. Li and Jurafsky (2015) show that sense embeddings can significantly improve performance of part-of-speech tagging, semantic relation identification and semantic relatedness tasks, but yield no improvement for named entity recognition and sentiment analysis. Sense …

Content truncated to 500 words