What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6064–6070, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6064 What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition Ting-Yun Chang Yun-Nung Chen Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan r06922168@ntu.edu.tw y.v.chen@ieee.org Abstract Contextualized word embeddings have boosted many NLP tasks compared with traditional static word embeddings. However, the word with a specific sense may have different contextualized embeddings due to its various contexts. To further investigate what contextualized word embeddings capture, this paper analyzes whether they can indicate the corresponding sense definitions and proposes a general framework that is capable of ex- plaining word meanings given contextualized word embeddings for better interpretation. The experiments show that both ELMo and BERT embeddings can be well interpreted via a readable textual form, and the findings may benefit the research community for a better understanding of what the embeddings capture1. 1 Introduction Contextualized word embeddings, such as ELMo, BERT, and OpenAI GPT, GPT-2 (Peters et al., 2018; Devlin et al., 2018; Radford et al., a,b) have been shown to yield richer representations of meaning and boosted many NLP tasks. To under- stand what contextualized word embeddings cap- ture, Schuster et al. (2019) recently visualized the representations of ELMo and showed that 1) em- beddings of the same word in different contexts can form a cluster, and 2) when a word has mul- tiple senses, the embeddings can be separated into multiple distinct groups, one for each meaning. To further investigate the meanings contextual- ized word embeddings indicate, this paper focuses on analyzing whether a contextualized embedding is sense-informative enough to indicate the cor- responding sense definition given a (target word, context) pair. We train and evaluate our model on 1The source code and the trained models are publicly available at https://github.com/MiuLab/GenDef. the online Oxford dictionary dataset released by Chang et al. (2018). To analyze if the embeddings are sense- informative, our work focuses on learning a map- ping between the semantic space of contextual- ized word embeddings and the space of definition embeddings. Specifically, a better mapping indi- cates richer sense-specific cues in the contextual- ized word embedding. Different from the definition modeling in the prior work (Noraset et al., 2017; Gadetsky et al., 2018; Chang et al., 2018), we reformulate the task from natural language generation (NLG) to clas- sification, i.e., selecting the most reasonable def- inition according to the target word and its con- texts. As recent work has shown the great suc- cess in encoding lexical resources into consis- tent representations (Tissier et al., 2017; Bahdanau et al., 2017; Bosc and Vincent, 2018), in this pa- per, we leverage pretrained sentence encoder (Cer et al., 2018) to project all definitions in the Oxford dictionary to a consistent embedding space, sup- porting our reformulation which requires to learn a mapping transforming from the contextualized word embedding …

Content truncated to 500 words