SemEval-2010 Task 1: Coreference Resolution in Multiple Languages

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 1–8, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 1: Coreference Resolution in Multiple Languages Marta Recasens⋆ Llu´ıs M`arquez† Emili Sapena† M. Ant`onia Mart´ı⋆ Mariona Taul´e⋆ V´eronique Hoste‡ Massimo Poesio⋄ Yannick Versley⋆⋆ ⋆: CLiC, University of Barcelona, {mrecasens,amarti,mtaule}@ub.edu †: TALP, Technical University of Catalonia, {lluism,esapena}@lsi.upc.edu ‡: University College Ghent, veronique.hoste@hogent.be ⋄: University of Essex/University of Trento, poesio@essex.ac.uk ⋆⋆: University of T¨ubingen, versley@sfs.uni-tuebingen.de Abstract This paper presents the SemEval-2010 task on Coreference Resolution in Multi- ple Languages. The goal was to evaluate and compare automatic coreference reso- lution systems for six different languages (Catalan, Dutch, English, German, Italian, and Spanish) in four evaluation settings and using four different metrics. Such a rich scenario had the potential to provide insight into key issues concerning corefer- ence resolution: (i) the portability of sys- tems across languages, (ii) the relevance of different levels of linguistic information, and (iii) the behavior of scoring metrics. 1 Introduction The task of coreference resolution, defined as the identification of the expressions in a text that re- fer to the same discourse entity (1), has attracted considerable attention within the NLP community. (1) Major League Baseball sent its head of se- curity to Chicago to review the second in- cident of an on-field fan attack in the last seven months. The league is reviewing se- curity at all ballparks to crack down on spectator violence. Using coreference information has been shown to be beneficial in a number of NLP applications including Information Extraction (McCarthy and Lehnert, 1995), Text Summarization (Steinberger et al., 2007), Question Answering (Morton, 1999), and Machine Translation. There have been a few evaluation campaigns on coreference resolution in the past, namely MUC (Hirschman and Chinchor, 1997), ACE (Doddington et al., 2004), and ARE (Orasan et al., 2008), yet many questions remain open: • To what extent is it possible to imple- ment a general coreference resolution system portable to different languages? How much language-specific tuning is necessary? • How helpful are morphology, syntax and se- mantics for solving coreference relations? How much preprocessing is needed? Does its quality (perfect linguistic input versus noisy automatic input) really matter? • How (dis)similar are different coreference evaluation metrics—MUC, B-CUBED, CEAF and BLANC? Do they all provide the same ranking? Are they correlated? Our goal was to address these questions in a shared task. Given six datasets in Catalan, Dutch, English, German, Italian, and Spanish, the task we present involved automatically detecting full coreference chains—composed of named entities (NEs), pronouns, and full noun phrases—in four different scenarios. For more information, the reader is referred to the task website.1 The rest of the paper is organized as follows. Section 2 presents the corpora from which the task datasets were extracted, and the automatic tools used to preprocess them. In Section 3, we describe the task by providing information about the data format, evaluation settings, and evaluation met- rics. Participating systems are described in Sec- tion …

Content truncated to 500 words