Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5297–5306, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5297 Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan Department of Computer Science, National University of Singapore christian.hadiwinoto@u.nus.edu, nght@comp.nus.edu.sg, gan weechung@u.nus.edu Abstract Contextualized word representations are able to give different representations for the same word in different contexts, and they have been shown to be effective in downstream natu- ral language processing tasks, such as ques- tion answering, named entity recognition, and sentiment analysis. However, evaluation on word sense disambiguation (WSD) in prior work shows that using contextualized word representations does not outperform the state- of-the-art approach that makes use of non- contextualized word embeddings. In this pa- per, we explore different strategies of integrat- ing pre-trained contextualized word represen- tations and our best strategy achieves accura- cies exceeding the best prior published accura- cies by significant margins on multiple bench- mark WSD datasets. 1 Introduction Word sense disambiguation (WSD) automatically assigns a pre-defined sense to a word in a text. Dif- ferent senses of a word reflect different meanings a word has in different contexts. Identifying the cor- rect word sense given a context is crucial in natural language processing (NLP). Unfortunately, while it is easy for a human to infer the correct sense of a word given a context, it is a challenge for NLP systems. As such, WSD is an important task and it has been shown that WSD helps downstream NLP tasks, such as machine translation (Chan et al., 2007a) and information retrieval (Zhong and Ng, 2012). A WSD system assigns a sense to a word by tak- ing into account its context, comprising the other words in the sentence. This can be done through discrete word features, which typically involve surrounding words and collocations trained using a classifier (Lee et al., 2004; Ando, 2006; Chan et al., 2007b; Zhong and Ng, 2010). The classifier can also make use of continuous word represen- tations of the surrounding words (Taghipour and Ng, 2015; Iacobacci et al., 2016). Neural WSD systems (K˚ageb¨ack and Salomonsson, 2016; Ra- ganato et al., 2017b) feed the continuous word rep- resentations into a neural network that captures the whole sentence and the word representation in the sentence. However, in both approaches, the word representations are independent of the context. Recently, pre-trained contextualized word rep- resentations (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019) have been shown to improve downstream NLP tasks. Pre-trained contextualized word represen- tations are obtained through neural sentence en- coders trained on a huge amount of raw texts. When the resulting sentence encoder is fine-tuned on the downstream task, such as question answer- ing, named entity recognition, and sentiment anal- ysis, with much smaller annotated training data, it has been shown that the trained model, with the …

Content truncated to 500 words