SENSEVAL-2 The Swedish Framework

SENSEV AL-2: The Swedish Framework Dimitrios KOKKINAKIS Sprakdata, Goteborg University Jerker JARBORG Sprakdata, Gateborg University Yvonne CEDERHOLM Sprakdata, Goteborg University Box 200, SE-405 30 Goteborg, Sweden Dimitrios.Kokkinakis @svenska.gu.se Box 200, SE-405 30 Goteborg, Sweden Jerker.Jaerborg@ svenska.gu.se Box 200, SE-405 30 Goteborg, Sweden Yvonne. Cederholm@ svenska.gu.se Abstract In this paper we describe the organisation and results of the SENSEVAL-2 exercise for Swedish. We present some of the experiences we gained by participating as developers and organisers in the exercise. We particularly focus on the choice of the lexical and corpus material, the annotation process, the scoring scheme, the motivations for choosing the lexical-sample branch of the exercise, the participating systems and the official results. Introduction Word sense ambiguity is a potential source for errors in human language technology applications, such as Machine Translation, and it is considered as the great open problem at the lexical level of Natural Language Processing (NLP). There are, however, several computer programs for automatically determining which sense of a word is being used in a given context, according to a variety of semantic, or defining dictionaries as demonstrated in the SENSEV AL-l exercise; (Kilgarriff and Palmer, 2000). The purpose of SENSEV AL is to be able to say which programs and methods perform better, which worse, which words, or varieties of language, present particular problems to which programs; when modifications improve performance of systems, and how much and what combinations of modifications are optimal. Specifically for Swedish, we would also like to investigate to what extent sense disambiguation can be accomplished and the potential resources available for the task. We would thus be creating a framework that can be shared both within the 45 exercise and for future evaluation exercises of similar kind, national and international. 1 Choice of Task Three tasks were identified for SENSEVAL-2, namely: the lexical-sample, the all-words and the 'in a system' tasks. In the lexical sample task, first, we sample the lexicon, then we find instances in context of the sample words and the evaluation . is carried out on the sampled instances. In the all-word task a system will be evaluated on its disambiguation performance on every word in the test collection. Finally, in the third type of task, a. word sense disambiguation (WSD) system is evaluated on how well it improves the performance of a NL system (MT, IR etc). The reasons we chose the lexical-sample task for Swedish are summarised below: 1. Cost-effectiveness of annotation: it is easier and quicker for the human annotators to sense-tag multiple occurrences of one word at a time, particularly when robust interactive means are utilized (Section 3); 2. The lexical-sample reduces the work of preparing training data since only a subset of the sense inventory is used; 3. More systems can/could (eventually) participate; 4. The all-words task requires access to a full dictionary, which is problematic from the copyright point of view, since industrial partners were also allowed to participate; and, as Kilgarriff and Palmer (2000) noted: 5. Provided that the sample …

Content truncated to 500 words