GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3509–3514, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3509 GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge Luyao Huang, Chi Sun, Xipeng Qiu∗, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {lyhuang18,sunc17,xpqiu,xjhuang}@fudan.edu.cn Abstract Word Sense Disambiguation (WSD) aims to find the exact sense of an ambiguous word in a particular context. Traditional supervised methods rarely take into consideration the lex- ical resources like WordNet, which are widely utilized in knowledge-based methods. Recent studies have shown the effectiveness of incor- porating gloss (sense definition) into neural networks for WSD. However, compared with traditional word expert supervised methods, they have not achieved much improvement. In this paper, we focus on how to better leverage gloss knowledge in a supervised neural WSD system. We construct context-gloss pairs and propose three BERT-based models for WSD. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task. 1 Introduction Word Sense Disambiguation (WSD) is a funda- mental task and long-standing challenge in Nat- ural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a par- ticular context (Navigli, 2009). Previous WSD ap- proaches can be grouped into two main categories: knowledge-based and supervised methods. Knowledge-based WSD methods rely on lex- ical resources like WordNet (Miller, 1995) and usually exploit two kinds of lexical knowledge. The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm (Lesk, 1986) and then widely taken into account in many other approaches (Banerjee and Pedersen, 2002; Basile et al., 2014). Besides, structural properties of se- mantic graphs are mainly used in graph-based al- gorithms (Agirre et al., 2014; Moro et al., 2014). Traditional supervised WSD methods (Zhong and Ng, 2010; Shen et al., 2013; Iacobacci et al., ∗Corresponding author. 2016) focus on extracting manually designed fea- tures and then train a dedicated classifier (word ex- pert) for every target lemma. Although word expert supervised WSD meth- ods perform better, they are less flexible than knowledge-based methods in the all-words WSD task (Raganato et al., 2017a). Recent neural-based methods are devoted to dealing with this prob- lem. K˚ageb¨ack and Salomonsson (2016) present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer. Raganato et al. (2017a) convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words. However, neither of them can totally beat the best word ex- pert supervised methods. More recently, Luo et al. (2018b) propose to leverage the gloss information from WordNet and model the semantic relationship between the con- text and gloss in an improved memory network. Similarly, Luo et al. (2018a) introduce a (hi- erarchical) co-attention mechanism to generate co-dependent representations for the context and gloss. Their attempts prove …

Content truncated to 500 words