VUA-background : When to Use Background Information to Perform Word Sense Disambiguation

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 345–349, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics VUA-background : When to Use Background Information to Perform Word Sense Disambiguation Marten Postma VU University Amsterdam Ruben Izquierdo VU University Amsterdam {m.c.postma, ruben.izquierdobevia, p.t.j.m.vossen}@vu.nl Piek Vossen VU University Amsterdam Abstract We present in this paper our submission to task 13 of SemEval2015, which makes use of background information and external re- sources (DBpedia and Wikipedia) to automat- ically disambiguate texts. Our approach fol- lows two routes for disambiguation: one route is proposed by a state–of–the–art WSD sys- tem, and the other one by the predominant sense information extracted in an unsuper- vised way from an automatically built back- ground corpus. We reached 4th position in terms of F1-score in task number 13 of Se- mEval2015: “Multilingual All-Words Sense Disambiguation and Entity Linking” (Moro and Navigli, 2015). All the software and code created for this approach are publicly available on GitHub1. 1 Introduction Word Sense Disambiguation is still an unsolved problem in Natural Language Processing. Many different approaches have been proposed through- out the years to tackle this task from different per- spectives. In addition, competitions have been or- ganized to compare the performance of these ap- proaches. Our hypothesis is that, in general, the context is not being modelled properly by the sys- tems, which usually consider very narrow contexts and do not pay any attention to the background in- formation or information that is not explicitly in- cluded in the text. We conducted an in-depth er- ror analysis of previous all-words tasks (Senseval–2 : English all words (Palmer et al., 2001), Senseval– 3 : English all words (Snyder and Palmer, 2004), Semeval–2007 : all words task 17 (Pradhan et al., 2007), Semeval–2010 : all words task 17 (Agirre et al., 2010), Semeval–2013 : all words task 12 (Nav- igli et al., 2013)) in order to gain better insight as to 1https://github.com/cltl/ vua-wsd-sem2015 why some approaches perform better than others, to detect problems not properly addressed and to try to overcome them. 2 We observed that most systems tend to rely on lo- cal features (words surrounding the words in ques- tion) to perform word sense disambiguation. Be- sides this, there is a very acute trend by all WSD systems to assign in most cases the most frequent sense, regardless the domain under consideration, as can be seen in Figure 1: sval2 sval3 sval2007 sval2013 0 20 40 60 80 100 Accuracy Accuracy when sense is MFS versus when it is not MFS NotMFS Figure 1: The average accuracy of all systems per com- petition is shown. Figure 1 shows the average accuracy of all the systems per competition. We clearly observe the trend that systems perform well when the sense is the most frequent sense, but not in other cases. Fur- thermore, when the sense is not the most frequent one, the systems still propose the most frequent sense. For instance in Senseval–2, out of …

Content truncated to 500 words