SemEval-2010 Task 3: Cross-Lingual Word Sense Disambiguation

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 15–20, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 3: Cross-Lingual Word Sense Disambiguation Els Lefever1,2 and Veronique Hoste1,2 1LT3, Language and Translation Technology Team, University College Ghent, Belgium 2Department of Applied Mathematics and Computer Science, Ghent University, Belgium {Els.Lefever,Veronique.Hoste}@hogent.be Abstract The goal of this task is to evaluate the feasibility of multilingual WSD on a newly developed multilingual lexi- cal sample data set. Participants were asked to automatically determine the contextually appropriate translation of a given English noun in five languages, viz. Dutch, German, Italian, Spanish and French. This paper reports on the sixteen submissions from the five dif- ferent participating teams. 1 Introduction Word Sense Disambiguation, the task of se- lecting the correct sense of an ambiguous word in a given context, is a well-researched NLP problem (see for example Agirre and Edmonds (2006) and Navigli (2009)), largely boosted by the various Senseval and SemEval editions. The SemEval-2010 Cross-lingual Word Sense Disambiguation task focuses on two bottle- necks in current WSD research, namely the scarcity of sense inventories and sense-tagged corpora (especially for languages other than English) and the growing tendency to eval- uate the performance of WSD systems in a real application such as machine translation and cross-language information retrieval (see for example Agirre et al. (2007)). The Cross-lingual WSD task aims at the de- velopment of a multilingual data set to test the feasibility of multilingual WSD. Many studies have already shown the validity of this cross- lingual evidence idea (Gale et al., 1993; Ide et al., 2002; Ng et al., 2003; Apidianaki, 2009), but until now no benchmark data sets have been available. For the SemEval-2010 compe- tition we developed (i) a sense inventory in which the sense distinctions were extracted from the multilingual corpus Europarl1 and (ii) a data set in which the ambiguous words were annotated with the senses from the mul- tilingual sense inventory. The Cross-Lingual WSD task is a lexical sample task for English nouns, in which the word senses are made up of the translations in five languages, viz. Dutch, French, Italian, Spanish and German. Both the sense inventory and the annotated data set were constructed for a sample of 25 nouns. The data set was divided into a trial set of 5 ambiguous nouns and a test set of 20 nouns. The participants had to automatically deter- mine the contextually appropriate translation for a given English noun in each or a subset of the five target languages. Only translations present in Europarl were considered as valid translations. The remainder of this article is organized as follows. Section 2 focuses on the task descrip- tion and gives a short overview of the construc- tion of the sense inventory and the annotation of the benchmark data set with the senses from the multilingual sense inventory. Section 3 clarifies the scoring metrics and presents two frequency-based baselines. The participating systems are presented in Section 4, while …

Content truncated to 500 words