Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8498–8508, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8498 Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations G´abor Berend1,2 1Institute of Informatics, University of Szeged 2MTA-SZTE Research Group on Aartificial Intelligence berendg@inf.u-szeged.hu Abstract In this paper, we demonstrate that by utiliz- ing sparse word representations, it becomes possible to surpass the results of more com- plex task-specific models on the task of fine- grained all-words word sense disambiguation. Our proposed algorithm relies on an overcom- plete set of semantic basis vectors that allows us to obtain sparse contextualized word repre- sentations. We introduce such an information theory-inspired synset representation based on the co-occurrence of word senses and non- zero coordinates for word forms which allows us to achieve an aggregated F-score of 78.8 over a combination of five standard word sense disambiguating benchmark datasets. We also demonstrate the general applicability of our proposed framework by evaluating it towards part-of-speech tagging on four different tree- banks. Our results indicate a significant im- provement over the application of the dense word representations. 1 Introduction Natural language processing applications have ben- efited remarkably form language modeling based contextualized word representations, including CoVe (McCann et al., 2017), ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), inter alia. Contrary to standard “static” word embeddings like word2vec (Mikolov et al., 2013) and Glove (Pen- nington et al., 2014), contextualized representa- tions assign such vectorial representations to men- tions of word forms that are sensitive to the entire sequence in which they are present. This charac- teristic of contextualized word embeddings makes them highly applicable for performing word sense disambiguation (WSD) as it has been investigated recently (Loureiro and Jorge, 2019; Vial et al., 2019). Another popular line of research deals with sparse overcomplete word representations which differ from typical word embeddings in that most coefficients are exactly zero. Such sparse word representations have been argued to convey an increased interpretability (Murphy et al., 2012; Faruqui et al., 2015; Subramanian et al., 2018) which could be advantageous for WSD. It has been shown that sparsity can not only favor interpretabil- ity, but it can contribute to an increased perfor- mance in downstream applications (Faruqui et al., 2015; Berend, 2017). The goal of this paper is to investigate and quan- tify what synergies exist between contextualized and sparse word representations. Our rigorous ex- periments show that it is possible to get increased performance on top of contextualized representa- tions when they are post-processed in a way which ensures their sparsity. In this paper we introduce an information theory- inspired algorithm for creating sparse contextu- alized word representations and evaluate it in a series of challenging WSD tasks. In our exper- iments, we managed to obtain solid results for multiple fine-grained word sense disambiguation benchmarks. All our source code for reproduc- ing our experiments are made available at https: //github.com/begab/sparsity_makes_sense.1 Our contributions can be summarized as follows: • we …

Content truncated to 500 words