SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 778–786, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering Anne-Lyse Minard1, Manuela Speranza1, Eneko Agirre2, Itziar Aldabe2, Marieke van Erp3, Bernardo Magnini1, German Rigau2, Rub´en Urizar2 1 Fondazione Bruno Kessler, Trento, Italy 2 The University of the Basque Country (UPV/EHU), Spain 3 VU University Amsterdam, the Netherlands {minard,manspera,magnini}@fbk.eu, marieke.van.erp@vu.nl {itziar.aldabe,e.agirre,german.rigau,ruben.urizar}@ehu.eus Abstract This paper describes the outcomes of the TimeLine task (Cross-Document Event Order- ing), that was organised within the Time and Space track of SemEval-2015. Given a set of documents and a set of target entities, the task consisted of building a timeline for each en- tity, by detecting, anchoring in time and or- dering the events involving that entity. The TimeLine task goes a step further than previ- ous evaluation challenges by requiring partic- ipant systems to perform both event corefer- ence and temporal relation extraction across documents. Four teams submitted the output of their systems to the four proposed subtracks for a total of 13 runs, the best of which ob- tained an F1-score of 7.85 in the main track (timeline creation from raw text). 1 Introduction In any domain, it is important that professionals have access to high quality knowledge for taking well- informed decisions. As daily tasks of information professionals revolve around reconstructing a chain of previous events, an insightful way of presenting information to them is by means of timelines. The aim of the Cross-Document Event Ordering task is to build timelines from English news articles. To provide focus to the timeline creation, the task is pre- sented as an ordering task in which events involving a particular target entity are to be ordered chrono- logically. The task focuses on cross-document event coreference resolution and cross-document temporal relation extraction. The latter has been the topic of the three previous TempEval tasks within the SemEval challenges: • TempEval-1 (2007): Temporal Relation Identifi- cation (Verhagen et al., 2009) • TempEval-2 (2010): Evaluating Events, Time Ex- pressions, and Temporal Relations (Verhagen et al., 2010) • TempEval-3 (2013): Temporal Annotation (Uz- Zaman et al., 2013) Additionally, it has also been the focus of the 6th i2b2 NLP Challenge for clinical records (Sun et al., 2013). The cross-document aspect, however, has not often been explored. One example is the work de- scribed in (Ji et al., 2009) using the ACE 2005 train- ing corpora. Here the authors link pre-defined events involving the same centroid entities (i.e. entities fre- quently participating in events) on a timeline. Nom- inal coreference resolution has been the topic of Se- mEval 2010 Task on Coreference Resolution in Mul- tiple Languages (Recasens et al., 2010). TimeLine is a pilot task that goes beyond the above-mentioned evaluation exercises by addressing coreference reso- lution for events and temporal relation extraction at a cross document level. This task was motivated by work done in the NewsReader project1. The goal of the NewsReader project is to reconstruct story lines …

Content truncated to 500 words