Quasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation

Proceedings of Recent Advances in Natural Language Processing, pages 122–131, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_015 122 Quasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation Michele Bevilacqua and Roberto Navigli Department of Computer Science Sapienza University of Rome {bevilacqua,navigli}@di.uniroma1.it Abstract While contextualized embeddings have produced performance breakthroughs in many Natural Language Processing (NLP) tasks, Word Sense Disambiguation (WSD) has not benefited from them yet. In this pa- per, we introduce QBERT, a Transformer- based architecture for contextualized em- beddings which makes use of a co- attentive layer to produce more deeply bidirectional representations, better-fitting for the WSD task. As a result, we are able to train a WSD system that beats the state of the art on the concatenation of all evaluation datasets by over 3 points, also outperforming a comparable model using ELMo. 1 Introduction Word Sense Disambiguation (WSD) is the task of associating a word in context with the right mean- ing among a finite set of possible senses (Navigli, 2009). Consider the following sentence, in which SERVED is the target word: (1) The waiter standing near the counter SERVED the revolutionary cause well. In WordNet (Fellbaum, 1998), the most used En- glish computational lexicon in NLP, the following two senses are associated (among many others) to the verb to serve: 1. devotion: devote (part of) one’s life or efforts to, as of countries, institutions, or ideas; 2. food: help to some food; help with some food or drink; The WSD system, in this case, would be tasked to associate the target word with the correct meaning – i.e. the devotion sense. Currently the best WSD systems are supervised, i.e. they leverage annotated corpora as training data (Yuan et al., 2016; Vial et al., 2018; Melacci et al., 2018). However, data labeling is a bot- tleneck for WSD, even more so than in other fields of NLP. Semantic annotation is a costly pro- cess, requiring expert annotators (Taghipour and Ng, 2015; Pasini and Navigli, 2017). If we con- sider that neural networks, the best performing ap- proach in virtually every task in NLP, are partic- ularly data-hungry, it appears unlikely that there will be much progress in WSD unless either more data is available, or less data is needed. Between the two directions, we believe efforts towards the latter will prove more fruitful, firstly, because of scalability considerations, and sec- ondly, and more importantly, because of the re- cent growth in the use of transfer learning, as ex- emplified by contextualized embeddings. Con- textualized embeddings have been shown to pro- duce much better results on downstream tasks compared to end-to-end training, even when less data is provided (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; He et al., 2018; Akbik et al., 2018). Contextualized embed- dings that use words as tokenization units, such as ELMo (Peters et al., 2018), are most suited to WSD. They are usually trained through self- supervised Causal Language Modeling (CLM) (Lample and Conneau, 2019): given a word …

Content truncated to 500 words