Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy Yi Zhou Department of Computer Science University of Liverpool, UK Y.Zhou71@liverpool.ac.uk Danushka Bollegala Department of Computer Science University of Liverpool, UK danushka@liverpool.ac.uk Abstract Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static word embeddings such as GloVe represent words by relatively low-dimensional, memory- and compute-efficient vectors but are not sensitive to the different senses of the word. We pro- pose Context Derived Embeddings of Senses (CDES), a method that extracts sense related in- formation from contextualised embeddings and injects it into static embeddings to create sense- specific static embeddings. Experimental re- sults on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sense- specific static embeddings reporting compara- ble performance to the current state-of-the-art sense embeddings. 1 Introduction Representing the meanings of words using low- dimensional vector embeddings has become a standard technique in NLP. Static word embed- dings (Mikolov et al., 2013; Pennington et al., 2014) represent words at the form level by assigning a single vector for all occurrences of a word irrespective of its senses. However, representing ambiguous words such as bass, which could mean either a musical in- strument or a type of fish, using a single embedding is problematic. To address this problem, sense-specific static word embedding methods (Reisinger and Mooney, 2010; Figure 1: Outline of CDES. Given a sense-tagged sen- tence t, we compute a sense embedding for the ambigu- ous word bank by multiplying its static word embedding, g(bank), by a sense-specific projection matrix, Abank%00, corresponding to the correct sense of the word. Projec- tion matrices are learnt by minimising the squared ℓ2 loss between the linearly transformed (via a matrix W) con- textualised embedding, c(t, bank), and of the (nonlinearly transformed via function f) sense embedding of bank. Neelakantan et al., 2014; Huang et al., 2012) assign multiple embeddings to a single polysemous word corresponding to its multiple senses. However, these embeddings are context-insensitive and we must re- sort to different heuristics such as selecting the sense embedding of the ambiguous word that is most sim- ilar to the context, to determine which embedding should be selected to represent the word. On the other hand, contextualised word embed- dings generated from NLMs (Peters et al., 2018a; Devlin et al., 2019; Liu et al., 2019) represent a word in a given context by an embedding that considers both the meaning of the word itself as well as its context. Different types of information such as word sense, dependency, and numeracy have shown to be encoded in contextualised word embeddings, provid- ing rich, context-sensitive input representations for numerous downstream NLP applications. More re- cently, Loureiro and Jorge (2019a) and Scarlini et al. (2020a) showed that contextualised embeddings such as BERT (Devlin et al., 2019) and ELMo (Pe- ters et al., …

Content truncated to 500 words