Revisiting the Evaluation for Cross Document Event Coreference

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949–1958, Osaka, Japan, December 11-17 2016. Revisiting the Evaluation for Cross Document Event Coreference Shyam Upadhyay Nitish Gupta Christos Christodoulopoulos Dan Roth {upadhya3,ngupta19,christod,danr}@illinois.edu University of Illinois at Urbana-Champaign, IL, USA Abstract Cross document event coreference (CDEC) is an important task that aims at aggregating event- related information across multiple documents. We revisit the evaluation for CDEC, and dis- cover that past works have adopted different, often inconsistent, evaluation settings, which either overlook certain mistakes in coreference decisions, or make assumptions that simplify the coref- erence task considerably. We suggest a new evaluation methodology which overcomes these limitations, and allows for an accurate assessment of CDEC systems. Our new evaluation setting better reflects the corpus-wide information aggregation ability of CDEC systems by separating event-coreference decisions made across documents from those made within a document. In addition, we suggest a better baseline for the task and semi-automatically identify several incon- sistent annotations in the evaluation dataset. 1 Introduction Understanding events is crucial to natural language understanding and has applications ranging from question answering (Berant et al., 2014; Narayanan and Harabagiu, 2004), to causal reasoning (Do et al., 2011; Chambers and Jurafsky, 2008) to headline generation (Sun et al., 2015). The task of Cross Document Event Coreference (CDEC) determines if two event mentions (which belong to different doc- uments) refer to the same event (Bagga and Baldwin, 1999; Bejan and Harabagiu, 2014). Figure 1 shows an example of two events whose mentions co-refer across 3 documents, [King]'decided'to'fire' [O’Brien]','who's;ll'has'two' years'lea'on'the'contract'he' signed'when'he'was'hired'by' the'[Sixers]'on'April'21','2004.' Up'the'coast,'the'[Philly'Sixers]' canned'[Jim'O’Brien]'('who'had' been'under'fire'all'season')'and' quickly'hired'[Mo'Cheeks].'' [76ers]'have'relieved'[Maurice' Cheeks]'of'his'coaching' responsibili;es'.'' [Cheeks]'was'brought1aaer'[Jim' O'Brien]'was'relieved'of'his'du;es.' O’Brien fired! Cheeks hired! Figure 1: Event coreference across 3 documents. Event mentions are shown in bold and its participants are enclosed by brackets, []. Note that there are multiple firing events, but only a few co-refer. An efficient CDEC system enables corpus-level aggregation of event attributes, which can prove valu- able for tasks such as information extraction and aggregation (Humphreys et al., 1997; Zhang et al., 2015), topic detection and tracking (Allan et al., 1998), multi-document summarization (Daniel et al., 2003) and knowledge discovery (Mayfield et al., 2009). In this work, we analyze whether existing CDEC evaluations reflect a system’s ability to predict cross document links. Past works have adopted different evaluation methodologies which either make simpli- fying assumptions about the coreference task, such as ignoring singletons, or overlook certain corefer- ence mistakes made by a CDEC system (as discussed in §4). Furthermore, under existing evaluations, a system which only predicts within document coreference links can score higher than a system which This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1949 predicts both within and cross document links. However, the latter system is clearly useful at aggregating information at corpus-level by finding across document links. We address these issues by lifting the aforementioned assumptions, and proposing a new setting which enables accurate evaluation of the cross document coreference performance (§ …

Content truncated to 500 words