Interpretable Word Embedding Contextualization

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 341–343 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 341 Interpretable Word Embedding Contextualization Kyoung-Rok Jang School of Computing KAIST Daejeon, South Korea kyoungrok.jang@kaist.ac.kr Sung-Hyon Myaeng School of Computing KAIST Daejeon, South Korea myaeng@kaist.ac.kr Sang-Bum Kim Naver Corp. Seongnam-si, South Korea sangbum.kim@navercorp.com Abstract In this paper, we propose a method of calibrat- ing a word embedding, so that the semantic it conveys becomes more relevant to the con- text. Our method is novel because the output shows clearly which senses that were origi- nally presented in a target word embedding be- come stronger or weaker. This is possible by utilizing the technique of using sparse coding to recover senses that comprises a word em- bedding. 1 Introduction In this paper we propose a method of generat- ing contextualized word embeddings. What we mean by ‘contextualized’ is that standard embed- dings such as Skip-gram and GloVe are modified to reflect their contexts. For instance, apple ap- peared in fruit-implying context should become more similar to fruit than it was in the prior state. We need contextualized embeddings because not all information contained in an embedding is helpful for modelling accurately the meaning of a word in context (e.g. company-related senses of apple in fruit-implying context). Since word em- beddings are trained on unconstrained variation of contexts, using word embeddings as-is is like tak- ing the risk of feeding our subsequent models (e.g. classifiers) with noises that are not relevant to the given context. We formulate our task as calibrating senses con- tained in embeddings so that contextually relevant senses (e.g. fruit-ness) becomes stronger and the others (e.g. company-ness) become weaker. To achieve this we utilize the technique of recovering standard word embeddings as linear composition of different senses, proposed by (Murphy et al., 2012; Faruqui et al., 2015; Arora et al., 2016). Af- ter applying the technique word embeddings are transformed into high dimensional (e.g. 2,500) and sparse (only small portion of dimensions are nonzero) embeddings. This makes our method interpretable since extracted senses can give us “a succinct description of which other words co- occur with a specific word sense”. More detailed explanation is presented in Section 3. Using the technique we first decompose word embeddings of a target word (to be contextual- ized) and context words into linear composition of senses, then identify strong senses extracted from context and regard them as contextually relevant. Finally we use the contextually relevant senses for calibrating the senses contained in a target word. 2 Task and Model 2.1 Task We show that our method is effective by applying it to Word Sense Discrimination (WSD) task. For brevity we present only one instance of sense dis- crimination: discriminating apple as a fruit or a company depending on given context. The result is described in Section 3 2.2 Embedding Decomposition To contextualize a target word’s embedding, we should first decompose the participating word em- beddings (i.e. target …

Content truncated to 500 words