Which Evaluations Uncover Sense Representations that Actually Make Sense?

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1727–1738 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1727 Which Evaluations Uncover Sense Representations that Actually Make Sense? Fenfei Guo, Jordan Boyd-Graber, Mohit Iyyer, Leah Findlater University of Maryland, University of Maryland, University of Massachussetts, University of Washington fenfeigo@cs.umd.edu, jbg@umiacs.umd.edu, miyyer@cs.umass.edu, leahkf@uw.edu Abstract Text representations are critical for modern natural language processing. One form of text representation, sense-specific embeddings, reflect a word’s sense in a sentence better than single-prototype word embeddings tied to each type. However, existing sense representations are not uniformly better: although they work well for computer-centric evaluations, they fail for human-centric tasks like inspecting a language’s sense inventory. To expose this discrepancy, we propose a new coherence evaluation for sense embeddings. We also describe a minimal model (Gumbel Attention for Sense Induction) optimized for discovering interpretable sense representations that are more coherent than existing sense embeddings. Keywords: sense representations, interpretability evaluation, embeddings 1. Context, Sense, and Representation Computers need to represent the meaning of words in context. BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018) have dramatically changed how natural language processing represents text. Rather than one-size-fits-all word vectors that ignore the nuance of how words are used in context, these new repre- sentations have topped the leaderboards for question answering, inference, and classification. Contextual representations have supplanted multisense embed- dings (Camacho-Collados and Pilehvar, 2018). While these methods learn a vector for each sense, they do not encode meanings in downstream tasks as well as contextual representations (Peters et al., 2018). However, computers are not the only consumer of text repre- sentations. Humans also use word representations to under- stand diachronic drift, investigate a language’s sense inven- tory, or to cluster and explore documents. Thus, a primary role for multisense word embeddings is human understand- ing of word meanings. Unfortunately, multisense models have only been evaluated on computer-centric dimensions and have ignored the question of sense interpretability. We first develop measures for how well models encode and explain a word’s meaning to a human (Section 4). Exist- ing multisense models do not necessarily fare best on this evaluation; our simpler model (Gumbel Attention for Sense Induction: GASI, Section 2) that focuses on discrete sense selection can better capture human-interpretable represen- tations of senses; comparing against traditional evaluations (Section 5), GASI has better contextual word similarity and competitive non-contextual word similarity. Finally, we dis- cuss the connections between representation learning and how modern contextual representations could better capture interpretable senses (Section 6). 2. Attentional Sense Induction Before we explore human interpretability of sense in- duction, we first describe our simple models to dis- entangle word senses. Our two models are built on Word2Vec (Mikolov et al., 2013a, Mikolov et al., 2013b), which we review in Section 2.1. Both models use a straight- forward attention mechanism to select which sense is used in a token’s context, which we contrast to alternatives for You only live twice, Mr. …

Content truncated to 500 words