On Coreference Resolution Performance Metrics

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 25–32, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics On Coreference Resolution Performance Metrics Xiaoqiang Luo 1101 Kitchawan Road, Room 23-121 IBM T.J. Wastson Research Center Yorktown Heights, NY 10598, U.S.A. xiaoluo@us.ibm.com Abstract The paper proposes a Constrained Entity- Alignment F-Measure (CEAF) for evaluating coreference resolution. The metric is com- puted by aligning reference and system entities (or coreference chains) with the constraint that a system (reference) entity is aligned with at most one reference (system) entity. We show that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm. Comparative exper- iments are conducted to show that the widely- known MUC F-measure has serious flaws in evaluating a coreference system. The proposed metric is also compared with the ACE-Value, the official evaluation metric in the Automatic Content Extraction (ACE) task, and we con- clude that the proposed metric possesses some properties such as symmetry and better inter- pretability missing in the ACE-Value. 1 Introduction A working definition of coreference resolution is parti- tioning the noun phrases we are interested in into equiv- alence classes, each of which refers to a physical entity. We adopt the terminologies used in the Automatic Con- tent Extraction (ACE) task (NIST, 2003a) and call each individual phrase a mention and equivalence class an en- tity. For example, in the following text segment, (1): “The American Medical Association voted yesterday to install the heir apparent as its president-elect, rejecting a strong, upstart challenge by a district doctor who argued that the nation’s largest physicians’ group needs stronger ethics and new leadership.” mentions are underlined, “American Medical Associa- tion”, “its” and “group” refer to the same organization (object) and they form an entity. Similarly, “the heir ap- parent” and “president-elect” refer to the same person and they form another entity. It is worth pointing out that the entity definition here is different from what used in the Message Understanding Conference (MUC) task (MUC, 1995; MUC, 1998) – ACE entity is called coreference chain or equivalence class in MUC, and ACE mention is called entity in MUC. An important problem in coreference resolution is how to evaluate a system’s performance. A good performance metric should have the following two properties: � Discriminativity: This refers to the ability to differ- entiate a good system from a bad one. While this criterion sounds trivial, not all performance metrics used in the past possess this property. � Interpretability: A good metric should be easy to in- terpret. That is, there should be an intuitive sense of how good a system is when a metric suggests that a certain percentage of coreference results are correct. For example, when a metric reports ��� or above correct for a system, we would expect that the vast majority of mentions are in right entities or corefer- ence chains. A widely-used metric is the link-based F-measure (Vi- lain et al., 1995) …

Content truncated to 500 words