The Italian lexical sample task at Senseval-3

The Italian Lexical Sample Task at SENSEVAL-3 Bernardo Magnini, Danilo Giampiccolo and Alessandro Vallin ITC-Irst, Istituto per la Ricerca Scientifica e Tecnologica Via Sommarive, 18 – 38050 Trento, Italy {magnini, giampiccolo, vallin}@itc.it Abstract The Italian lexical sample task SENSEVAL-3 provided a framework toat evaluate supervised and semi-supervised WSD systems. This paper reports on the task preparation – which offered the op- portunity to review and refine the Italian MultiWordNet – and on the results of the six participants, focussing on both the manual and automatic tagging procedures. 1 Introduction The task consisted in automatically determining the correct meaning of a word within a given con- text (i.e. a short text snippet). Systems’ results were compared on the one hand to those achieved by human annotators (upper bound), and on the other hand to those returned by a basic algorithm (baseline). In the second section of this paper an overview of the task preparation is given and in the follow- ing one the main features of the participating sys- tems are briefly outlined and the results of the evaluation exercise are presented. In the conclusions we give an overall judgement of the outcome of the task, suggesting possible im- provements for the next campaign. 2 Manual Annotation A collection of manually labeled instances was 1.built for three main reasons: automatic evaluation (using the Scorer2 pro- gram) required a Gold Standard list of senses provided by human annotators; 2. supervised WSD systems need a labeled set of training data, that in our case was twice larger than the test set; 3. manual semantic annotation is a time- consuming activity, but SENSEVAL repre- sents the framework to build reusable bench- mark resources. Besides, manual sense tagging entails the revision of the sense inventory, whose granularity does not always satisfy an- notators. 2.1 Corpus and Words Choice The document collection from which the anno- tators selected the text snippets containing the lemmata to disambiguate was the macro-balanced section of the Meaning Italian Corpus (Bentivogli et al., 2003). This corpus is an open domain col- lection of newspaper articles that contains about 90 million tokens covering a time-spam of 4 years (1998-2001). The corpus was indexed in order to browse it with the Toolbox for Lexicographers (Giuliano, 2002), a concordancer that enables tag- gers to highlight the occurrences of a token within a context. Two taggers chose 45 lexical entries (25 nouns, 10 adjectives and 10 verbs) according to their polysemy in the sense inventory, their polysemy in the corpus and their frequency (Edmonds, 2000). The words that had already been used SENSEVAL-2 were avoided. Ten words wereat shared with the Spanish, Catalan and Basque lexi- cal sample tasks. Annotators were provided with a formula that indicated the number of labeled instances for each lemma1, so they checked that the words were con- 1 No. of labeled instances for each lemma = 75 + (15*no. of attested senses) + (7* no. of attested multiwords), where 75 is a fixed number of examples distrib- uted over …

Content truncated to 500 words