Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 656–664, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art Veselin Stoyanov Cornell University Ithaca, NY ves@cs.cornell.edu Nathan Gilbert University of Utah Salt Lake City, UT ngilbert@cs.utah.edu Claire Cardie Cornell University Ithaca, NY cardie@cs.cornell.edu Ellen Riloff University of Utah Salt Lake City, UT riloff@cs.utah.edu Abstract We aim to shed light on the state-of-the-art in NP coreference resolution by teasing apart the differ- ences in the MUC and ACE task definitions, the as- sumptions made in evaluation methodologies, and inherent differences in text corpora. First, we exam- ine three subproblems that play a role in coreference resolution: named entity recognition, anaphoric- ity determination, and coreference element detec- tion. We measure the impact of each subproblem on coreference resolution and confirm that certain as- sumptions regarding these subproblems in the eval- uation methodology can dramatically simplify the overall task. Second, we measure the performance of a state-of-the-art coreference resolver on several classes of anaphora and use these results to develop a quantitative measure for estimating coreference resolution performance on new data sets. 1 Introduction As is common for many natural language process- ing problems, the state-of-the-art in noun phrase (NP) coreference resolution is typically quantified based on system performance on manually anno- tated text corpora. In spite of the availability of several benchmark data sets (e.g. MUC-6 (1995), ACE NIST (2004)) and their use in many formal evaluations, as a field we can make surprisingly few conclusive statements about the state-of-the- art in NP coreference resolution. In particular, it remains difficult to assess the ef- fectiveness of different coreference resolution ap- proaches, even in relative terms. For example, the 91.5 F-measure reported by McCallum and Well- ner (2004) was produced by a system using perfect information for several linguistic subproblems. In contrast, the 71.3 F-measure reported by Yang et al. (2003) represents a fully automatic end-to-end resolver. It is impossible to assess which approach truly performs best because of the dramatically different assumptions of each evaluation. Results vary widely across data sets. Corefer- ence resolution scores range from 85-90% on the ACE 2004 and 2005 data sets to a much lower 60- 70% on the MUC 6 and 7 data sets (e.g. Soon et al. (2001) and Yang et al. (2003)). What accounts for these differences? Are they due to properties of the documents or domains? Or do differences in the coreference task definitions account for the dif- ferences in performance? Given a new text collec- tion and domain, what level of performance should we expect? We have little understanding of which aspects of the coreference resolution problem are handled well or poorly by state-of-the-art systems. Ex- cept for some fairly general statements, for exam- ple that proper names are easier to resolve than pronouns, which are easier than common nouns, there has been little analysis of which aspects of the problem have achieved …

Content truncated to 500 words