Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1423–1432, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Coreference Resolution across Corpora: Languages, Coding Schemes, and Preprocessing Information Marta Recasens CLiC - University of Barcelona Gran Via 585 Barcelona, Spain mrecasens@ub.edu Eduard Hovy USC Information Sciences Institute 4676 Admiralty Way Marina del Rey CA, USA hovy@isi.edu Abstract This paper explores the effect that dif- ferent corpus configurations have on the performance of a coreference resolution system, as measured by MUC, B3, and CEAF. By varying separately three param- eters (language, annotation scheme, and preprocessing information) and applying the same coreference resolution system, the strong bonds between system and cor- pus are demonstrated. The experiments reveal problems in coreference resolution evaluation relating to task definition, cod- ing schemes, and features. They also ex- pose systematic biases in the coreference evaluation metrics. We show that system comparison is only possible when corpus parameters are in exact agreement. 1 Introduction The task of coreference resolution, which aims to automatically identify the expressions in a text that refer to the same discourse entity, has been an in- creasing research topic in NLP ever since MUC-6 made available the first coreferentially annotated corpus in 1995. Most research has centered around the rules by which mentions are allowed to corefer, the features characterizing mention pairs, the algo- rithms for building coreference chains, and coref- erence evaluation methods. The surprisingly im- portant role played by different aspects of the cor- pus, however, is an issue to which little attention has been paid. We demonstrate the extent to which a system will be evaluated as performing differ- ently depending on parameters such as the corpus language, the way coreference relations are de- fined in the corresponding coding scheme, and the nature and source of preprocessing information. This paper unpacks these issues by running the same system—a prototype entity-based architec- ture called CISTELL—on different corpus config- urations, varying three parameters. First, we show how much language-specific issues affect perfor- mance when trained and tested on English and Spanish. Second, we demonstrate the extent to which the specific annotation scheme (used on the same corpus) makes evaluated performance vary. Third, we compare the performance using gold- standard preprocessing information with that us- ing automatic preprocessing tools. Throughout, we apply the three principal coref- erence evaluation measures in use today: MUC, B3, and CEAF. We highlight the systematic prefer- ences of each measure to reward different config- urations. This raises the difficult question of why one should use one or another evaluation mea- sure, and how one should interpret their differ- ences in reporting changes of performance score due to ‘secondary’ factors like preprocessing in- formation. To this end, we employ three corpora: ACE (Doddington et al., 2004), OntoNotes (Pradhan et al., 2007), and AnCora (Recasens and Mart´ı, 2009). In order to isolate the three parameters as far as possible, we benefit from a 100k-word portion (from the TDT collection) that is common to both …

Content truncated to 500 words