SemEval-2007 Task 09: Multilevel Semantic Annotation of Catalan and Spanish

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 42–47, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 09: Multilevel Semantic Annotation of Catalan and Spanish Llu´ıs M`arquez and Luis Villarejo TALP Research Center Technical University of Catalonia {lluism,luisv}@lsi.upc.edu M. A. Mart´ı and Mariona Taul´e Centre de Llenguatge i Computaci´o, CLiC Universitat de Barcelona {amarti,mtaule}@ub.edu Abstract In this paper we describe SemEval-2007 task number 9 (Multilevel Semantic Annotation of Catalan and Spanish). In this task, we aim at evaluating and comparing automatic systems for the annotation of several seman- tic linguistic levels for Catalan and Spanish. Three semantic levels are considered: noun sense disambiguation, named entity recogni- tion, and semantic role labeling. 1 Introduction The Multilevel Semantic Annotation of Catalan and Spanish task is split into the following three sub- tasks: Noun Sense Disambiguation (NSD): Disambigua- tion of all frequent nouns (“all words” style). Named Entity Recognition (NER): The annotation of (possibly embedding) named entities with basic entity types. Semantic Role Labeling (SRL): Including also two subtasks, i.e., the annotation of verbal predicates with semantic roles (SR), and verb tagging with semantic–class labels (SC). All semantic annotation tasks are performed on exactly the same corpora for each language. We pre- sented all the annotation levels together as a com- plex global task, since we were interested in ap- proaches which address these problems jointly, pos- sibly taking into account cross-dependencies among them. However, we were also accepting systems ap- proaching the annotation in a pipeline style, or ad- dressing any of the particular subtasks in any of the languages. In Section 2 we describe the methodology fol- lowed to develop the linguistic corpora for the task. Sections 3 and 4 summarize the task setting and the participant systems, respectively. Finally, Section 5 presents a comparative analysis of the results. For any additional information on corpora, resources, formats, tagsets, annotation manuals, etc. we refer the reader to the official website of the task1. 2 Linguistic corpora The corpora used in this SemEval task are a subset of CESS-ECE, a multilingual Treebank, composed of a Spanish (CESS-ESP) and a Catalan (CESS-CAT) corpus of 500K words each (Mart´ı et al., 2007b). These corpora were enriched with different kinds of semantic information: argument structure, thematic roles, semantic class, named entities, and WordNet synsets for the 150 most frequent nouns. The an- notation process was carried out in a semiautomatic way, with a posterior manual revision of all auto- matic processes. A sequential approach was adopted for the anno- tation of the corpus, beginning with the basic lev- els of analysis, i.e., POS tagging and chunking (au- tomatically performed) and followed by the more complex levels: syntactic constituents and functions (manually tagged) and semantic annotation (man- ual and semiautomatic processes with manual com- pletion and posterior revision). Furthermore, some experiments concerning inter-annotator agreement 1www.lsi.upc.edu/∼nlp/semeval/msacs.html 42 were carried out at the syntactic (Civit et al., 2003) and semantic levels (M`arquez et al., 2004) in order to evaluate the quality of the results. …

Content truncated to 500 words