Singletons and Coreference Resolution Evaluation

Proceedings of Recent Advances in Natural Language Processing, pages 261–267, Hissar, Bulgaria, 12-14 September 2011. Singletons and Coreference Resolution Evaluation Sandra K¨ubler Indiana University skuebler@indiana.edu Desislava Zhekova University of Bremen zhekova@uni-bremen.de Abstract This paper presents an empirical study on the influence of singletons on the evalua- tion of coreference resolution systems. We present results on two English data sets used in the SEMEVAL 2010 shared task 1 and the CONLL 2011 shared task using the scorers of both shared tasks. We show that single- tons, both in the gold standard and in the sys- tem output, have an immense impact on the overall evaluation – in an experiment where the coreference resolution results remain un- changed over the different settings. 1 Introduction In the last decade, the task of Coreference Resolu- tion has become an important enterprise in Natural Language Processing. At the same time, the need for proper benchmarking increased over time. In the last year, two major shared tasks were concerned with coreference resolution: the SEMEVAL 2010 task 1 “Coreference Resolution in Multiple Languages” (Recasens et al., 2010) and the CONLL shared task 2011 “Modeling Unrestricted Coreference in OntoNotes” (Pradhan et al., 2011). Both shared tasks introduced a new element into the definition of coreference resolution: The detection of men- tions. Previous to these shared tasks, the availability of gold standard mentions was often assumed, and research concentrated on the resolution of corefer- ence relationships between mentions. (e.g. (Luo et al., 2004; Denis and Baldridge, 2007)). However, in many approaches to coreference res- olution, the problem is even more restricted, and the coreference resolution component expects only such mentions that are coreferent in the present context, i.e. no singletons are present in the data. “Sin- gleton” is a cover term for mentions that are never coreferent, such as in in general or on the contrary, and mentions that are potentially coref- erent but occur only once in a document. If the ex- traction of mentions is part of the task definition, then filtering singletons is generally necessary since methods for mention identification often overgener- ate and produce all noun phrases (NPs), including all singletons. Twinless mentions (Stoyanov et al., 2009) are mentions that have been identified by a coreference resolution system but are not included in the gold data, or vice versa. Twinless mentions can lead to considerable changes in overall system performance, and Stoyanov et al. (2009) report that at that time B3 was not prepared to handle them. For the CONLL shared task, the metrics were updated to obtain ”better alignment for B3 and CEAF so that the gold standard set and the system output have the same number of mentions” (p.c. S. Pradhan). In this paper, we investigate how the presence of singletons in either gold standard or in the system output influ- ences the results. We compare the English data sets of the SEMEVAL and the CONLL shared task and the two versions of the scorer used there. A simple solution was chosen by …

Content truncated to 500 words