Cross-Document, Cross-Language Event Coreference Annotation Using Event Hoppers

Cross-Document, Cross-Language Event Coreference Annotation Using Event Hoppers Zhiyi Song, Ann Bies, Justin Mott, Xuansong Li, Stephanie Strassel, Christopher Caruso Linguistic Data Consortium Suite 810, 3600 Market Street, Philadelphia, Pennsylvania, 19104, USA {zhiyi, bies, jmott, xuansong, strassel, ccaruso}@ldc.upenn.edu Abstract We discuss the development and implementation of an approach for cross-document, cross-lingual event coreference for the DEFT Rich Entities, Relations and Events (Rich ERE) annotation task. Rich ERE defined the notion of event hoppers to enable intuitive within- document coreference for the DEFT event ontology, and the expansion of coreference to cross-document, cross-lingual event mentions relies crucially on this same construct. We created new annotation guidelines, data processes and user interfaces to enable annotation of 505 documents in three languages selected from data already labeled for Rich ERE, yielding 389 cross-document event hoppers. We discuss the data creation process and the central role of event hoppers in making cross-document, cross-lingual coreference decisions. We present the challenges encountered during annotation along with three directions for future work. Keywords: Events, Information Extraction, Coreference, Linguistic Resources 1. Introduction This paper discusses the development and implementation of annotation for cross-document and cross-lingual event coreference as an expansion to the Rich Entities, Relations, and Events (ERE) annotation task first defined as part of DARPA’s Deep Exploration and Filtering of Text (DEFT) program (DARPA, 2012). The goal of the DEFT program is to develop technologies capable of extracting knowledge from unstructured text in multiple languages and genres. ERE annotation was developed at Linguistic Data Consortium (LDC) to support multiple research directions and evaluations in DEFT. ERE builds on the approach to labeling entities, relations, events and their attributes under a pre-defined taxonomy, following the approach used in Automatic Content Extraction (ACE) (LDC, 2005; Walker et al., 2006; Song et al., 2015; Mott et al., 2016). One important DEFT use case is automatically building a structured Knowledge Base (KB) from scratch. This task, known as Cold Start, is one of several tasks relevant to DEFT that are evaluated in the NIST Knowledge Base Population evaluation series (NIST 2017; Getman et al, 2018). Given DEFT’s focus on whole-corpus understanding culminating in the Cold Start task, Rich ERE annotation has evolved over the course of the program to emphasize cross-document and cross-lingual approaches. Rich ERE event annotation includes 9 event types and 38 subtypes (e.g. Conflict.Attack, Contact.Meet, Movement.TransportPerson). For each event mention annotators label the most salient word evoking the event (the “event trigger”), the event type and subtype, the realis status (Actual, Other or Generic), all of the event’s arguments (e.g. agent, instrument) and several attributes like temporal information (Aguilar et al., 2014; Song et al., 2015). Multiple mentions of the same event in the document are labeled for coreference, utilizing the notion of event hoppers. While stricter approaches to event coreference typically require all event features (including arguments and attributes) to be identical, event hoppers relax this requirement to enable coreference of two events that are intuitively the same although certain features may differ. For instance, two reports …

Content truncated to 500 words