SemEval-2013 Task 10: Cross-lingual Word Sense Disambiguation

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 158–166, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 10: Cross-lingual Word Sense Disambiguation Els Lefever1,2 and V´eronique Hoste1,3 1LT3, Language and Translation Technology Team, University College Ghent Groot-Brittanni¨elaan 45, 9000 Gent, Belgium 2Department of Applied Mathematics, Computer Science and Statistics, Ghent University Krijgslaan 281 (S9), 9000 Gent, Belgium 3Department of Linguistics, Ghent University Blandijnberg 2, 9000 Gent, Belgium {Els.Lefever,Veronique.Hoste}@hogent.be Abstract The goal of the Cross-lingual Word Sense Disam- biguation task is to evaluate the viability of multilin- gual WSD on a benchmark lexical sample data set. The traditional WSD task is transformed into a mul- tilingual WSD task, where participants are asked to provide contextually correct translations of English ambiguous nouns into five target languages, viz. French, Italian, English, German and Dutch. We re- port results for the 12 official submissions from 5 different research teams, as well as for the ParaSense system that was developed by the task organizers. 1 Introduction Lexical ambiguity remains one of the major prob- lems for current machine translation systems. In the following French sentence “Je cherche des id´ees pour manger de l’avocat”1, the word “avocat” is clearly referring to the fruit, whereas both Google Translate2 as well as Babelfish3 translate the word as “lawyer”. Although “lawyer” is a correct transla- tion of the word “avocat”, it is the wrong translation in this context. Other language technology applica- tions, such as Question Answering (QA) systems or information retrieval (IR) systems, also suffer from the poor contextual disambiguation of word senses. Word sense disambiguation (WSD) is still consid- ered one of the most challenging problems within 1English translation: “I’m looking for ideas to eat avocado”. 2http://translate.google.com 3http://be.bing.com/translator/ language technology today. It requires the construc- tion of an artificial text understanding as the sys- tem should detect the correct word sense based on the context of the word. Different methodologies have been investigated to solve the problem; see for instance Agirre and Edmonds (2006) and Navigli (2009) for a detailed overview of WSD algorithms and evaluation. This paper reports on the second edition of the “Cross-Lingual Word Sense Disambiguation” (CLWSD) task, that builds further on the insights we gained from the SemEval-2010 evaluation (Lefever and Hoste, 2010b) and for which new test data were annotated. The task is an unsupervised Word Sense Disambiguation task for English nouns, the sense label of which is composed of translations in dif- ferent target languages (viz. French, Italian, Span- ish, Dutch and German). The sense inventory is built up on the basis of the Europarl parallel corpus; all translations of a polysemous word were manu- ally grouped into clusters, which constitute different senses of that given word. For the test data, native speakers assigned a translation cluster(s) to each test sentence and gave their top three translations from the predefined list of Europarl translations, in order to assign weights to the set of gold standard …

Content truncated to 500 words