A Deep Dive into Word Sense Disambiguation with LSTM

Proceedings of the 27th International Conference on Computational Linguistics, pages 354–365 Santa Fe, New Mexico, USA, August 20-26, 2018. 354 A Deep Dive into Word Sense Disambiguation with LSTM Minh Le†, Marten Postma†, Jacopo Urbani‡ and Piek Vossen† † Department of Language, Literature and Communication, Vrije Universiteit Amsterdam ‡ Department of Computer Science, Vrije Universiteit Amsterdam {m.n.le,m.c.postma,piek.vossen}@vu.nl, jacopo@cs.vu.nl Abstract LSTM-based language models have been shown effective in Word Sense Disambiguation (WSD). In particular, the technique proposed by Yuan et al. (2016) returned state-of-the-art per- formance in several benchmarks, but neither the training data nor the source code was released. This paper presents the results of a reproduction study and analysis of this technique using only openly available datasets (GigaWord, SemCor, OMSTI) and software (TensorFlow). Our study showed that similar results can be obtained with much less data than hinted at by Yuan et al. (2016). Detailed analyses shed light on the strengths and weaknesses of this method. First, adding more unannotated training data is useful, but is subject to diminishing returns. Second, the model can correctly identify both popular and unpopular meanings. Finally, the limited sense coverage in the annotated datasets is a major limitation. All code and trained models are made freely available. 1 Introduction Word Sense Disambiguation (WSD) is a long-established task in the NLP community (see Navigli (2009) for a survey) which goal is to annotate lemmas in text with the most appropriate meaning from a lexical database like WordNet (Fellbaum, 1998). Many approaches have been proposed – the more popular ones include the usage of Support Vector Machine (SVM) (Zhong and Ng, 2010), SVM combined with unsupervised trained embeddings (Iacobacci et al., 2016; Rothe and Sch¨utze, 2017), and graph-based approaches (Agirre et al., 2014; Weissenborn et al., 2015). In recent years, there has been a surge in interest in using Long short-term memory (LSTM) (Hochre- iter and Schmidhuber, 1997) to perform WSD (Raganato et al., 2017b; Melamud et al., 2016). These approaches are characterized by their high performance, simplicity and their ability to extract a lot of information from raw text. Among the best-performing ones is the approach by Yuan et al. (2016), in which an LSTM language model trained on a corpus with 100 billion tokens was coupled with small sense-annotated datasets to achieve state-of-the-art performance in all-words WSD. Even though the results obtained by Yuan et al. (2016) outperform the previous state-of-the-art, nei- ther the used datasets nor the constructed models are available to the community. This is unfortunate because this makes the re-application of this technique a non-trivial process, and it hinders further stud- ies for understanding which limitations prevent even higher accuracies. These could be, for instance, of algorithmic nature or relate to the input (either size or quality), and a deeper understanding is crucial for enabling further improvements. In addition, some details are not reported, and this could prevent other attempts from replicating the results. To address these issues, we reimplemented Yuan et al. (2016)’s method with the goal of: 1) …

Content truncated to 500 words