SemEval-2007 Task 07: Coarse-Grained English All-Words Task

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 30–35, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 07: Coarse-Grained English All-Words Task Roberto Navigli Universit`a di Roma “La Sapienza” Dipartimento di Informatica Via Salaria, 00198 - Roma Italy navigli@di.uniroma1.it Kenneth C. Litkowski CL Research 9208 Gue Road Damascus MD 20872 ken@clres.com Orin Hargraves Lexicographer orinhargraves @googlemail.com Abstract This paper presents the coarse-grained En- glish all-words task at SemEval-2007. We describe our experience in producing a coarse version of the WordNet sense inven- tory and preparing the sense-tagged corpus for the task. We present the results of par- ticipating systems and discuss future direc- tions. 1 Introduction It is commonly thought that one of the major obsta- cles to high-performance Word Sense Disambigua- tion (WSD) is the fine granularity of sense inven- tories. State-of-the-art systems attained a disam- biguation accuracy around 65% in the Senseval-3 all-words task (Snyder and Palmer, 2004), where WordNet (Fellbaum, 1998) was adopted as a ref- erence sense inventory. Unfortunately, WordNet is a fine-grained resource, encoding sense distinctions that are difficult to recognize even for human an- notators (Edmonds and Kilgarriff, 2002). Making WSD an enabling technique for end-to-end applica- tions clearly depends on the ability to deal with rea- sonable sense distinctions. The aim of this task was to explicitly tackle the granularity issue and study the performance of WSD systems on an all-words basis when a coarser set of senses is provided for the target words. Given the need of the NLP community to work on freely available resources, the solution of adopting a dif- ferent computational lexicon is not viable. On the other hand, the production of a coarse-grained sense inventory is not a simple task. The main issue is certainly the subjectivity of sense clusters. To overcome this problem, different strategies can be adopted. For instance, in the OntoNotes project (Hovy et al., 2006) senses are grouped until a 90% inter-annotator agreement is achieved. In contrast, as we describe in this paper, our approach is based on a mapping to a previously existing inventory which encodes sense distinctions at different levels of granularity, thus allowing to induce a sense clus- tering for the mapped senses. We would like to mention that another SemEval- 2007 task dealt with the issue of sense granularity for WSD, namely Task 17 (subtask #1): Coarse- grained English Lexical Sample WSD. In this paper, we report our experience in organizing Task 07. 2 Task Setup The task required participating systems to annotate open-class words (i.e. nouns, verbs, adjectives, and adverbs) in a test corpus with the most appropriate sense from a coarse-grained version of the WordNet sense inventory. 2.1 Test Corpus The test data set consisted of 5,377 words of run- ning text from five different articles: the first three (in common with Task 17) were obtained from the WSJ corpus, the fourth was the Wikipedia entry for computer programming1, the fifth was an excerpt of Amy Steedman’s Knights of the Art, biographies of …

Content truncated to 500 words