SENSEVAL-2: Overview

SENSEV AL-2: Overview Philip Edmonds Sharp Laboratories of Europe Oxford Science Park Oxford OX4 4GB, UK phil@ sharp.co.uk Abstract SENSEV AL-2: The Second International Workshop on Evaluating Word Sense Disambiguation Systems was held on July 5-6, 2001. This paper gives an overview of SENSEV AL-2, discussing the evaluation exercise, the tasks, the scoring system, and the results. It ends with some recommendations for future evaluation exercises. 1 Introduction Word sense disambiguation (WSD) is the problem of automatically deciding which sense a word has in any particular context. The success of any project in WSD is clearly tied to the evaluation of WSD systems. SENSEV AL was started in 1997, under the auspices of ACL- SIGLEX, to bring together researchers to discuss and solve the WSD-evaluation problem. Its aim is to evaluate the strengths and weaknesses of WSD algorithms and systems with respect to different words, different varieties of language, and different languages. SENSEV AL is independent from other evaluation programs in the language technology community, such as TREC and MUC. Unlike these programs, SENSEV AL is a 'freelance' program is run entirely by volunteers. We'd like to remind everyone that while SENSEV AL takes the guise of a competition, its main function is not to determine a winner but to explore the scientific aspects of word sense disambiguation. SENSEV AL held its first evaluation exercise in the summer of 1998, culminating in a workshop at Herstmonceux Castle, England on September 2-4 (Kilgarriff and Palmer 2000). Following the success of the first workshop, SENSEV AL-2, supported by EURALEX, 1 Scott Cotton Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA cotton@ linc.cis.upenn.edu ELSNET, EPSRC, and ELRA, was organized in 2000-2001. The Second International Workshop on Evaluating Word Sense Disambiguation Systems was held in conjunction with ACL-2001 on July 5-6, 2001 in Toulouse. This paper gives an overview of SENSEV AL- 2, discussing the evaluation exercise, the tasks, the scoring system, and the results. It ends with some recommendations for future evaluation exercises. 2 Tasks and participants A main goal of SENSEV AL-2 was to encourage new languages to participate. We were successful: SENSEV AL-2 evaluated WSD systems on three types of task on 12 languages as follows: All-words Czech, Dutch, English, Estonian Lexical Basque, English, Italian, sample Japanese, Korean, Spanish, Swedish Translation Japanese In the all-words task, systems must tag almost all of the content words in a sample of running text. In the lexical sample task, we first carefully select a sample of words from the lexicon; systems must then tag several instances of the sample words in short extracts of text. The translation task (Japanese only) is a lexical sample task in which word sense is defined according to translation distinction. Task design is discussed in section 3 below. 93 systems were submitted from 34 different research teams. Table 1 gives a breakdown of the number of submissions and teams who participated in each task. Note that some teams submitted multiple systems to the same task, and …

Content truncated to 500 words