SemEval-2013 Task 12: Multilingual Word Sense Disambiguation

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 222–231, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 12: Multilingual Word Sense Disambiguation Roberto Navigli, David Jurgens and Daniele Vannella Dipartimento di Informatica Sapienza Universit`a di Roma Viale Regina Elena, 295 – 00161 Roma Italy {navigli,jurgens,vannella}@di.uniroma1.it Abstract This paper presents the SemEval-2013 task on multilingual Word Sense Disambiguation. We describe our experience in producing a mul- tilingual sense-annotated corpus for the task. The corpus is tagged with BabelNet 1.1.1, a freely-available multilingual encyclopedic dictionary and, as a byproduct, WordNet 3.0 and the Wikipedia sense inventory. We present and analyze the results of participating sys- tems, and discuss future directions. 1 Introduction Word Sense Disambiguation (WSD), the task of au- tomatically assigning predefined meanings to words occurring in context, is a fundamental task in com- putational lexical semantics (Navigli, 2009; Navigli, 2012). Several Senseval and SemEval tasks have been organized in the past to study the performance and limits of disambiguation systems and, even more importantly, disambiguation settings. While an ad-hoc sense inventory was originally chosen for the first Senseval edition (Kilgarriff, 1998; Kilgarriff and Palmer, 2000), later tasks (Edmonds and Cot- ton, 2001; Snyder and Palmer, 2004; Mihalcea et al., 2004) focused on WordNet (Miller et al., 1990; Fellbaum, 1998) as a sense inventory. In 2007 the issue of the fine sense granularity of WordNet was addressed in two different SemEval disambiguation tasks, leading to the beneficial creation of coarser- grained sense inventories from WordNet itself (Nav- igli et al., 2007) and from OntoNotes (Pradhan et al., 2007). In recent years, with the exponential growth of the Web and, consequently, the increase of non- English speaking surfers, we have witnessed an up- surge of interest in multilinguality. SemEval-2010 tasks on cross-lingual Word Sense Disambiguation (Lefever and Hoste, 2010) and cross-lingual lexi- cal substitution (Mihalcea et al., 2010) were orga- nized. While these tasks addressed the multilin- gual aspect of sense-level text understanding, they departed from the traditional WSD paradigm, i.e., the automatic assignment of senses from an existing inventory, and instead focused on lexical substitu- tion (McCarthy and Navigli, 2009). The main factor hampering traditional WSD from going multilingual was the lack of a freely-available large-scale multi- lingual dictionary. The recent availability of huge collaboratively- built repositories of knowledge such as Wikipedia has enabled the automated creation of large-scale lexical knowledge resources (Hovy et al., 2013). Over the past few years, a wide-coverage multi- lingual “encyclopedic” dictionary, called BabelNet, has been developed (Navigli and Ponzetto, 2012a). BabelNet1 brings together WordNet and Wikipedia and provides a multilingual sense inventory that cur- rently covers 6 languages. We therefore decided to put the BabelNet 1.1.1 sense inventory to the test and organize a traditional Word Sense Disambigua- tion task on a given English test set translated into 4 other languages (namely, French, German, Spanish and Italian). Not only does BabelNet enable mul- tilinguality, but it also provides coverage …

Content truncated to 500 words