Recall Error Analysis for Coreference Resolution

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2070–2081, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Recall Error Analysis for Coreference Resolution Sebastian Martschat and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH Schloss-Wolfsbrunnenweg 35, 69118 Heidelberg, Germany (sebastian.martschat|michael.strube)@h-its.org Abstract We present a novel method for coreference resolution error analysis which we apply to perform a recall error analysis of four state-of-the-art English coreference reso- lution systems. Our analysis highlights differences between the systems and iden- tifies that the majority of recall errors for nouns and names are shared by all sys- tems. We characterize this set of com- mon challenging errors in terms of a broad range of lexical and semantic properties. 1 Introduction Coreference resolution is the task of determining which mentions in a text refer to the same entity. State-of-the-art approaches include both learning- based (Fernandes et al., 2012; Bj¨orkelund and Farkas, 2012; Durrett and Klein, 2013) and de- terministic models (Lee et al., 2013; Martschat, 2013). These approaches achieve state-of-the-art performance mainly relying on morphosyntactic and lexical factors. However, consider the follow- ing example. In order to improving the added value of oil products, the second phase project of the Qinghai Petroleum Bureau’s Ge’ermu oil refinery has been put into production. This will further improve the factory’s oil products structure. Due to the lack of any string overlap, most state-of-the-art systems will miss the link between the factory and the Qinghai Petroleum Bureau’s Ge’ermu oil refinery. The information that factory is a hypernym of refinery, however, may be useful to resolve such links. The aim of this paper is to quantify and char- acterize such recall errors made by state-of-the- art coreference resolution systems. By doing so, we provide a solid foundation for work on em- ploying knowledge sources for improving recall for coreference resolution (Ponzetto and Strube, 2006; Rahman and Ng, 2011; Ratinov and Roth, 2012; Bansal and Klein, 2012, inter alia). In par- ticular, we make the following contributions: We present a novel framework for coreference resolution error analysis. This yields a formal foundation for previous work on link-based error analysis (Uryupina, 2008; Martschat, 2013) and complements work on transformation-based error analysis (Kummerfeld and Klein, 2013). We apply the method proposed in this paper to perform a recall error analysis of four state-of- the-art systems, encompassing deterministic and learning-based approaches. In particular, we iden- tify and characterize a set of challenging errors common to all systems, and discuss strengths and weaknesses of each system regarding specific er- ror types. We also present a brief precision error analysis. A toolkit which implements the framework pro- posed in this paper is available for download.1 2 A Link-Based Analysis Framework In this section we discuss challenges in corefer- ence resolution error analysis and devise an error analysis framework to overcome these challenges. 2.1 Motivation Suppose a document contains the entity BARACK OBAMA, which is referenced by four mentions in the following order: Obama, he, the president and his. …

Content truncated to 500 words