SenseBERT: Driving Some Sense into BERT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4656–4667 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4656 SenseBERT: Driving Some Sense into BERT Yoav Levine Barak Lenz Or Dagan Ori Ram Dan Padnos Or Sharir Shai Shalev-Shwartz Amnon Shashua Yoav Shoham AI21 Labs, Tel Aviv, Israel {yoavl,barakl,ord,orir,...}@ai21.com Abstract The ability to learn from large unlabeled cor- pora has allowed neural language models to advance the frontier in natural language under- standing. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underly- ing semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet su- persenses. Accordingly, we attain a lexical- semantic level language model, without the use of human annotation. SenseBERT achieves sig- nificantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the ‘Word in Context’ task. 1 Introduction Neural language models have recently undergone a qualitative leap forward, pushing the state of the art on various NLP tasks. Together with advances in network architecture (Vaswani et al., 2017), the use of self-supervision has proven to be central to these achievements, as it allows the network to learn from massive amounts of unannotated text. The self-supervision strategy employed in BERT (Devlin et al., 2019) involves masking some of the words in an input sentence, and then training the model to predict them given their context. Other proposed approaches for self-supervised objectives, including unidirectional (Radford et al., 2019), per- mutational (Yang et al., 2019), or word insertion- based (Chan et al., 2019) methods, operate simi- larly, over words. However, since a given word form can possess multiple meanings (e.g., the word ‘bass’ can refer to a fish, a guitar, a type of singer, etc.), the word itself is merely a surrogate of its actual meaning in a given context, referred to as its sense. Indeed, the word-form level is viewed as a surface level which often introduces challenging ambiguity (Navigli, 2009). In this paper, we bring forth a novel method- ology for applying weak-supervision directly on the level of a word’s meaning. By infusing word- sense information into BERT’s pre-training sig- nal, we explicitely expose the model to lexical semantics when learning from a large unanno- tated corpus. We call the resultant sense-informed model SenseBERT. Specifically, we add a masked- word sense prediction task as an auxiliary task in BERT’s pre-training. Thereby, jointly with the stan- dard word-form level language model, we train a semantic-level language model that predicts the missing word’s meaning. Our method does not require sense-annotated data; self-supervised learn- ing from unannotated text is facilitated by using WordNet (Miller, 1998), an expert constructed in- ventory of word senses, as weak supervision. We focus on a coarse-grained variant of a word’s sense, referred to as its …

Content truncated to 500 words