Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006–1017 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1006 Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders Terra Blevins and Luke Zettlemoyer Paul G. Allen School of Computer Science & Engineering, University of Washington Facebook AI Research, Seattle {blvns, lsz}@cs.washington.edu Abstract A major obstacle in Word Sense Disambigua- tion (WSD) is that word senses are not uni- formly distributed, causing existing models to generally perform poorly on senses that are ei- ther rare or unseen during training. We pro- pose a bi-encoder model that independently embeds (1) the target word with its surround- ing context and (2) the dictionary definition, or gloss, of each sense. The encoders are jointly optimized in the same representation space, so that sense disambiguation can be performed by finding the nearest sense embedding for each target word embedding. Our system out- performs previous state-of-the-art models on English all-words WSD; these gains predom- inantly come from improved performance on rare senses, leading to a 31.1% error reduction on less frequent senses over prior work. This demonstrates that rare senses can be more ef- fectively disambiguated by modeling their def- initions. 1 Introduction One of the major challenges of Word Sense Disam- biguation (WSD) is overcoming the data sparsity that stems from the Zipfian distribution of senses in natural language (Kilgarriff, 2004). For example, in SemCor (the largest manually annotated dataset for WSD) 90% of mentions of the word plant cor- respond to the top two senses of the word, and only half of the ten senses of plant occur in the dataset at all (Miller et al., 1993). Due to this data imbalance, many WSD systems show a strong bias towards pre- dicting the most frequent sense (MFS) of a word regardless of the surrounding context (Postma et al., 2016). A successful WSD system should be able to over- come this bias and correctly disambiguate cases where a word takes a less frequent sense (LFS), without sacrificing performance on MFS examples. Previous work has found that incorporating lexical information such as sense definitions, or glosses, into WSD systems improves performance (Luo et al., 2018a,b).1 Glosses have also been found to improve LFS performance; however, absolute performance on rare senses is still low, with models showing a 62.3 F1 performance drop between the MFS examples and the LFS ones (Kumar et al., 2019). In this paper, we show that this gap can be signif- icantly reduced by jointly fine-tuning multiple pre- trained encoders on WSD. We present a bi-encoder model built on top of BERT (Devlin et al., 2019) that is designed to improve performance on rare and zero-shot senses. Similar to prior work, our system represents the target words and senses in the same embedding space by using a context en- coder to represent the target word and surrounding context, and a gloss encoder to represent the sense definitions. However, our two encoders are jointly …

Content truncated to 500 words