With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3528–3539, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3528 With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation Bianca Scarlini, Tommaso Pasini and Roberto Navigli Sapienza NLP Group Department of Computer Science Sapienza University of Rome {scarlini,pasini,navigli}@di.uniroma1.it Abstract Contextualized word embeddings have been employed effectively across several tasks in Natural Language Processing, as they have proved to carry useful semantic information. However, it is still hard to link them to struc- tured sources of knowledge. In this paper we present ARES (context-AwaRe Embeddings of Senses), a semi-supervised approach to pro- ducing sense embeddings for the lexical mean- ings within a lexical knowledge base that lie in a space that is comparable to that of contextu- alized word vectors. ARES representations en- able a simple 1-Nearest-Neighbour algorithm to outperform state-of-the-art models, not only in the English Word Sense Disambiguation task, but also in the multilingual one, whilst training on sense-annotated data in English only. We further assess the quality of our em- beddings in the Word-in-Context task, where, when used as an external source of knowledge, they consistently improve the performance of a neural model, leading it to compete with other more complex architectures. ARES em- beddings for all WordNet concepts and the automatically-extracted contexts used for cre- ating the sense representations are freely avail- able at http://sensembert.org/ares. 1 Introduction Contextualized word embeddings have proved to be highly beneficial to the majority of Natural Lan- guage Processing tasks (Wang et al., 2019). Indeed, language models like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), etc., enable architectures built on top of them to attain performances that were previously out of reach (Wang et al., 2019). The main reason behind this great success is the fact that contextu- alized embeddings of words encode the semantics defined by their input context (Reif et al., 2019). Indeed, when tested in the Word-in-Context (WiC) task (Pilehvar and Camacho-Collados, 2019), i.e., a binary classification problem where a model has to classify whether a target word is used with the same meaning in two different sentences, contex- tualized word embeddings placed themselves as the best approaches across the board. Nevertheless, these latent representations do not provide any ex- plicit information regarding the meaning expressed by the word in context, hence making it difficult to link texts to structured sources of knowledge such as lexical knowledge bases (LKB). The task of associating a word in context with the most suitable meaning from a predefined sense in- ventory is better known as Word Sense Disambigua- tion (Navigli, 2009, WSD), and is usually tackled by two kinds of approach: knowledge-based and supervised ones. On the one hand, knowledge- based approaches (Scozzafava et al., 2020; Conia and Navigli, 2020) are able to scale across lan- guages since they do not need sense-annotated cor- pora and rely only on the information within their underlying LKB. On …

Content truncated to 500 words