Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4884–4893 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4884 Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes Anastasia Zhukova1, Felix Hamborg2, Bela Gipp1,3 1 University of Wuppertal Gaußstraße 20, 42119 Wuppertal, Germany {last}@uni-wuppertal.de 2 University of Konstanz Universit¨atsstraße 10, 78464 Konstanz, Germany felix.hamborg@uni-konstanz.de 3 University of G¨ottingen Wilhelmsplatz 1, 37073 G¨ottingen, Germany gipp@cs.uni-goettingen.de Abstract Established cross-document coreference resolution (CDCR) datasets contain event-centric coreference chains of events and entities with identity relations. These datasets establish strict definitions of the coreference relations across related tests but typically ignore anaphora with more vague context-dependent loose coreference relations. In this paper, we qualitatively and quantitatively compare the annotation schemes of ECB+, a CDCR dataset with identity coreference relations, and NewsWCL50, a CDCR dataset with a mix of loose context-dependent and strict coreference relations. We propose a phrasing diversity metric (PD) that encounters for the diversity of full phrases unlike the previously proposed metrics and allows to evaluate lexical diversity of the CDCR datasets in a higher precision. The analysis shows that coreference chains of NewsWCL50 are more lexically diverse than those of ECB+ but annotating of NewsWCL50 leads to the lower inter-coder reliability. We discuss the different tasks that both CDCR datasets create for the CDCR models, i.e., lexical disambiguation and lexical diversity. Finally, to ensure generalizability of the CDCR models, we propose a direction for CDCR evaluation that combines CDCR datasets with multiple annotation schemes that focus of various properties of the coreference chains. Keywords: cross-document coreference resolution, lexical diversity, lexical disambiguation 1. Introduction Cross-document coreference resolution (CDCR) is a set of techniques that aims to resolve mentions of events and entities across a set of related documents. CDCR is employed as an essential analysis component in a broad spectrum of use cases, e.g., to identify potential targets in sentiment analysis or as a part of discourse interpretation. Although CDCR research has gained attention, the an- notation schemes and their corresponding datasets have only infrequently explored the mix of identity and lose coreference relations and lexical diversity of the anno- tated chains of mentions. For example, resolution of identity relations, i.e., coreference resolution, and reso- lution of more lose relations, i.e., bridging, are typically split into two separate tasks (Kobayashi and Ng, 2020). Resolution and evaluation of entity and event mentions of the two types of relations remain a research gap in CDCR research. This paper explores the qualitative and quantitative characteristics of two CDCR annotation schemes that annotate both events and entities. (1) ECB+, a state- of-the-art event-centric CDCR dataset (Cybulska and Vossen, 2014b) that annotates mentions with identity coreference relations. (2) NewsWCL50, an experimen- tal concept-centric dataset for identification of seman- tic concepts, each containing mentions with a mix of identity, loose coreference, and bridging relations, e.g., context-dependent anaphora, synonyms, metonyms, and meronyms (Hamborg et al., 2019). We propose a phras- ing diversity metric (PD) that describes …

Content truncated to 500 words