SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 1–6, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval Eneko Agirre IXA NLP group University of the Basque Country Donostia, Basque Counntry e.agirre@ehu.es Oier Lopez de Lacalle IXA NLP group University of the Basque Country Donostia, Basque Country jibloleo@ehu.es German Rigau IXA NLP group University of the Basque Country Donostia, Basque Country german.rigau@ehu.es Bernardo Magnini ITC-IRST Trento, Italy magnini@itc.it Arantxa Otegi IXA NLP group University of the Basque Country Donostia, Basque Country jibotusa@ehu.es Piek Vossen Irion Technologies Delftechpark 26 2628XH Delft, Netherlands Piek.Vossen@irion.nl Abstract This paper presents a first attempt of an application-driven evaluation exercise of WSD. We used a CLIR testbed from the Cross Lingual Evaluation Forum. The ex- pansion, indexing and retrieval strategies where fixed by the organizers. The par- ticipants had to return both the topics and documents tagged with WordNet 1.6 word senses. The organization provided training data in the form of a pre-processed Semcor which could be readily used by participants. The task had two participants, and the orga- nizer also provide an in-house WSD system for comparison. 1 Introduction Since the start of Senseval, the evaluation of Word Sense Disambiguation (WSD) as a separate task is a mature field, with both lexical-sample and all-words tasks. In the first case the participants need to tag the occurrences of a few words, for which hand-tagged data has already been provided. In the all-words task all the occurrences of open-class words occurring in two or three documents (a few thousand words) need to be disambiguated. The community has long mentioned the neces- sity of evaluating WSD in an application, in order to check which WSD strategy is best, and more im- portant, to try to show that WSD can make a differ- ence in applications. The use of WSD in Machine Translation has been the subject of some recent pa- pers, but less attention has been paid to Information Retrieval (IR). With this proposal we want to make a first try to define a task where WSD is evaluated with respect to an Information Retrieval and Cross-Lingual Infor- mation Retrieval (CLIR) exercise. From the WSD perspective, this task will evaluate all-words WSD systems indirectly on a real task. From the CLIR perspective, this task will evaluate which WSD sys- tems and strategies work best. We are conscious that the number of possible con- figurations for such an exercise is very large (in- cluding sense inventory choice, using word sense in- duction instead of disambiguation, query expansion, WSD strategies, IR strategies, etc.), so this first edi- tion focuses on the following: • The IR/CLIR system is fixed. • The expansion / translation strategy is fixed. • The participants can choose the best WSD strategy. 1 • The IR system is used as the upperbound for the CLIR systems. We think that it is important to start doing this kind of application-driven evaluations, which might shed light to the …

Content truncated to 500 words