Error-Driven Analysis of Challenges in Coreference Resolution

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 265–277, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Error-Driven Analysis of Challenges in Coreference Resolution Jonathan K. Kummerfeld and Dan Klein Computer Science Division University of California, Berkeley Berkeley, CA 94720, USA {jkk,klein}@cs.berkeley.edu Abstract Coreference resolution metrics quantify errors but do not analyze them. Here, we consider an automated method of categorizing errors in the output of a coreference system into intu- itive underlying error types. Using this tool, we first compare the error distributions across a large set of systems, then analyze common errors across the top ten systems, empirically characterizing the major unsolved challenges of the coreference resolution task. 1 Introduction Metrics produce measurements that concisely sum- marize performance on the full range of error types, and for coreference resolution there has been ex- tensive work on developing effective metrics (Luo, 2005; Recasens and Hovy, 2011). However, it is also valuable to tease apart the errors to understand their relative importance. Previous investigations of coreference errors have focused on quantifying the importance of subtasks such as named entity recognition and anaphoricity detection, typically by measuring accuracy improve- ments when partial gold annotations are provided (Stoyanov et al., 2009; Pradhan et al., 2011; Prad- han et al., 2012). For coreference resolution the drawback of this approach is that decisions are often interdependent, and so even partial gold information is extremely informative. Also, previous work only considered errors by counting links, which does not capture certain errors in a natural way, e.g. when a system incorrectly divides a large entity into two parts, each with multiple mentions. Recent work has considered some of these issues, but only with small scale manual analysis (Holen, 2013). We present a new tool that automatically classifies errors in the standard output of any coreference res- olution system. Our approach is to identify changes that convert the system output into the gold annota- tions, and map the steps in the conversion onto lin- guistically intuitive error types. Since our tool uses only system output, we are able to classify errors made by systems of any architecture, including both systems that use link-based inference and systems that use global inference methods. Using our tool we perform two studies to un- derstand similarities and differences between sys- tems. First, we compare the error distributions on coreference resolution of all of the systems from the CoNLL 2011 shared task plus several publicly avail- able systems. This comparison adds to the analy- sis from the shared task by illustrating the substan- tial variation in the types of errors different systems make. Second, we investigate the aggregate behav- ior of ten state-of-the-art systems, providing a de- tailed characterization of each error type. This in- vestigation identifies key outstanding challenges and presents the impact that solving each of them would have in terms of changes in the standard coreference resolution metrics. We find that the best systems are not best across all error types, …

Content truncated to 500 words