Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682–5691 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5682 Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation Daniel Loureiro, Al´ıpio M´ario Jorge LIAAD - INESC TEC Faculty of Sciences - University of Porto, Portugal dloureiro@fc.up.pt, amjorge@fc.up.pt Abstract Contextual embeddings represent a new gener- ation of semantic representations learned from Neural Language Modelling (NLM) that ad- dresses the issue of meaning conflation ham- pering traditional word embeddings. In this work, we show that contextual embeddings can be used to achieve unprecedented gains in Word Sense Disambiguation (WSD) tasks. Our approach focuses on creating sense-level embeddings with full-coverage of WordNet, and without recourse to explicit knowledge of sense distributions or task-specific modelling. As a result, a simple Nearest Neighbors (k- NN) method using our representations is able to consistently surpass the performance of pre- vious systems using powerful neural sequenc- ing models. We also analyse the robustness of our approach when ignoring part-of-speech and lemma features, requiring disambiguation against the full sense inventory, and revealing shortcomings to be improved. Finally, we ex- plore applications of our sense embeddings for concept-level analyses of contextual embed- dings and their respective NLMs. 1 Introduction Word Sense Disambiguation (WSD) is a core task of Natural Language Processing (NLP) which consists in assigning the correct sense to a word in a given context, and has many potential ap- plications (Navigli, 2009). Despite breakthroughs in distributed semantic representations (i.e. word embeddings), resolving lexical ambiguity has re- mained a long-standing challenge in the field. Sys- tems using non-distributional features, such as It Makes Sense (IMS, Zhong and Ng, 2010), remain surprisingly competitive against neural sequence models trained end-to-end. A baseline that simply chooses the most frequent sense (MFS) has also proven to be notoriously difficult to surpass. Several factors have contributed to this limited progress over the last decade, including lack of standardized evaluation, and restricted amounts of sense annotated corpora. Addressing the eval- uation issue, Raganato et al. (2017a) has intro- duced a unified evaluation framework that has al- ready been adopted by the latest works in WSD. Also, even though SemCor (Miller et al., 1994) still remains the largest manually annotated cor- pus, supervised methods have successfully used label propagation (Yuan et al., 2016), semantic networks (Vial et al., 2018) and glosses (Luo et al., 2018b) in combination with annotations to advance the state-of-the-art. Meanwhile, task- specific sequence modelling architectures based on BiLSTMs or Seq2Seq (Raganato et al., 2017b) haven’t yet proven as advantageous for WSD. Until recently, the best semantic representations at our disposal, such as word2vec (Mikolov et al., 2013) and fastText (Bojanowski et al., 2017), were bound to word types (i.e. distinct tokens), con- verging information from different senses into the same representations (e.g. ‘play song’ and ‘play tennis’ share the same representation of ‘play’). These word embeddings were learned from un- supervised Neural Language Modelling (NLM) trained on fixed-length …

Content truncated to 500 words