Linguistically Aware Coreference Evaluation Metrics

International Joint Conference on Natural Language Processing, pages 1366–1374, Nagoya, Japan, 14-18 October 2013. Linguistically Aware Coreference Evaluation Metrics Chen Chen and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {yzcchen,vince}@hlt.utdallas.edu Abstract Virtually all the commonly-used evalua- tion metrics for entity coreference reso- lution are linguistically agnostic, treating the mentions to be clustered as generic rather than linguistic objects. We argue that the performance of an entity coref- erence resolver cannot be accurately re- flected when it is evaluated using linguis- tically agnostic metrics. Consequently, we propose a framework for incorporating linguistic awareness into commonly-used coreference evaluation metrics. 1 Introduction Coreference resolution is the task of determin- ing which mentions in a text or dialogue refer to the same real-world entity. Designing appro- priate evaluation metrics for coreference resolu- tion is an important and challenging task. Since there is no consensus on which existing corefer- ence evaluation metric is the best, the organizers of the CoNLL-2011 and CoNLL-2012 shared tasks on unrestricted coreference (Pradhan et al., 2011, 2012) decided to take the average of the scores computed by three coreference evaluation metrics, MUC (Vilain et al., 1995), B3 (Bagga and Bald- win, 1998), and CEAFe (Luo, 2005), as the official score of a participating coreference resolver. One weakness shared by virtually all exist- ing coreference evaluation metrics is that they are linguistically agnostic, treating the mentions to be clustered as generic rather than linguistic objects. In other words, while MUC, B3, and CEAF were designed for evaluating coreference resolvers, their linguistic agnosticity implies that they can be used to evaluate any clustering task, including those that are not linguistic in nature.1 1This statement is also true for BLANC (Recasens and Hovy, 2011), a Rand Index-based coreference evaluation metric we will not focus on in this paper. To understand why linguistic agnosticity is a potential weakness of existing scoring metrics, consider a document in which there are three coreferent mentions, Hillary Clinton, she, and she, appearing in this order in the document. Assume that two coreference resolvers, R1 and R2, are applied to these three mentions, where R1 only posits Hillary Clinton and she as coreferent, and R2 only posits the two occurrences of she as coref- erent. Being linguistically agnostic, existing scor- ing metrics will assign the same score to both re- solvers after seeing that both of them correctly as- sign two of the three objects to the same cluster. Intuitively, however, R1 should receive a higher score than R2: R1 has facilitated automated text understanding by successfully finding the referent of one of the pronouns, whereas from R2’s output we know nothing about the referent of the two pro- nouns. Failure to rank R1 higher than R2 implies that existing scoring metrics fail to adequately re- flect the performance of a resolver.2 Our goal in this paper is to address the afore- mentioned weakness by proposing a framework for incorporating linguistic awareness into the most commonly-used coreference scoring metrics, including MUC, …

Content truncated to 500 words