Retrofitting Word Representations for Unsupervised Sense Aware Word Similarities

Retrofitting Word Representations for Unsupervised Sense Aware Word Similarities Steffen Remus and Chris Biemann Universität Hamburg Hamburg, Germany {remus,biemann}@informatik.uni-hamburg.de Abstract Standard word embeddings lack the possibility to distinguish senses of a word by projecting them to exactly one vector. This has a negative effect particularly when computing similarity scores between words using standard vector-based similarity measures such as cosine similarity. We argue that minor senses play an important role in word similarity computations, hence we use an unsupervised sense inventory resource to retrofit monolingual word embeddings, producing sense-aware embeddings. Using retrofitted sense-aware embeddings, we show improved word similarity and relatedness results on multiple word embeddings and multiple established word similarity tasks, sometimes up to an impressive margin of +0.15 Spearman correlation score. Keywords: word senses, word similarity and relatedness, word sense induction 1. Introduction Word embeddings – generated with neural networks (NN) or other factorization techniques – are a standard element in natural language processing (NLP) applications. How- ever, an important issue is their lack of sense-awareness, i.e. a word and its vector share a bijective mapping and a po- tential multiplicity of word meanings is ignored. The word iron, for example, which may refer to an atomic element, a device for smoothing clothes, a golf club, a color, or other meanings, is represented by a single common vector in the vector space. WORDNET (Fellbaum, 1998), on the other hand, defines four different interpretations of the word iron, and even this can never be considered to be complete as language evolves. Assigning the same vector for each dis- tinguished sense and using them in downstream tasks such as sentiment analysis, named entity recognition, question answering or many others, is error prone by design due to obvious misinterpretations and error propagation. Sense inventories – of which WORDNET is probably the most well known – are required to distinguish between different word senses, and meanings, rather than words, should be represented in the vector space (Navigli, 2009; Denkowski and Lavie, 2014). We use a simple, yet effec- tive technique to retrofit standard word embeddings to pro- duce embedding vectors of senses using external resources as sense inventories. Our hypothesis is that retrofitting pre- trained word embeddings to gain sense-aware embeddings is beneficial for word similarity computations. Using vec- tors of senses rather that vectors of words, we are indeed able to report substantial relative improvements for multi- ple word similarity tasks and for various types of retrofitted embeddings from five monolingual corpora. Because a word maps to multiple sense vectors in this sce- nario, standard cosine similarity computations alone are not applicable anymore, we thus test a number of sense-aware comparison methodologies based on cosine similarity. In particular for word pairs involving minor/rare senses, we expect improvements in the sense-aware setting as the in- fluence of the dominating major sense is diminished. Ad- ditionally, we compare our approach with two baseline ap- proaches to supervised and unsupervised sense-aware em- beddings: AUTOEXTEND (Rothe and Schütze, 2015) and ADAGRAM (Bartunov et al., 2016). To …

Content truncated to 500 words