Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 632–642, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric Nafise Sadat Moosavi and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH Schloss-Wolfsbrunnenweg 35 69118 Heidelberg, Germany {nafise.moosavi|michael.strube}@h-its.org Abstract Interpretability and discriminative power are the two most basic requirements for an evaluation metric. In this paper, we re- port the mention identification effect in the B3, CEAF, and BLANC coreference eval- uation metrics that makes it impossible to interpret their results properly. The only metric which is insensitive to this flaw is MUC, which, however, is known to be the least discriminative metric. It is a known fact that none of the current metrics are reliable. The common practice for rank- ing coreference resolvers is to use the av- erage of three different metrics. However, one cannot expect to obtain a reliable score by averaging three unreliable metrics. We propose LEA, a Link-based Entity-Aware evaluation metric that is designed to over- come the shortcomings of the current eval- uation metrics. LEA is available as branch LEA-scorer in the reference implemen- tation of the official CoNLL scorer. 1 Introduction There exists a variety of models (e.g. pairwise, entity-based, and ranking) and feature sets (e.g. string match, lexical, syntactic, and semantic) to be used in coreference resolution. There is no known formal way to prove which coreference model is superior to the others and which set of features is more beneficial/less useful in corefer- ence resolution. The only way to compare differ- ent models, features or implementations of coref- erence resolvers is to compare the values of the existing coreference resolution evaluation metrics. By comparing the evaluation scores, we determine which system performs best, which model suits coreference resolution better, and which feature set is useful for improving the recall or precision of a coreference resolver. Therefore, evaluation metrics play an important role in the advancement of the underlying technology. It is imperative for the evaluation metrics to be reliable. However, it is not a trivial task to score output entities with various kinds of coreference errors. Several evaluation metrics have been introduced for coreference resolution (Vilain et al., 1995; Bagga and Baldwin, 1998; Luo, 2005; Recasens and Hovy, 2011; Tuggener, 2014). Metrics that are being used widely are MUC (Vilain et al., 1995), B3 (Bagga and Baldwin, 1998), CEAF (Luo, 2005), and BLANC (Recasens and Hovy, 2011). There are known flaws for each of these metrics. Besides, the agreement between all these metrics is relatively low (Holen, 2013), and it is not clear which metric is the most reliable. The CoNLL-2011/2012 shared tasks (Pradhan et al., 2011; Pradhan et al., 2012) ranked participating systems using an average of three metrics, i.e. MUC, B3, and CEAF, following a proposal by (Denis and Baldridge, 2009a). Averaging three unreliable scores does not result in a reliable one. Besides, when an average score is used for …

Content truncated to 500 words