SemEval-2007 Task 15: TempEval Temporal Relation Identification

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 75–80, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 15: TempEval Temporal Relation Identification Marc Verhagen†, Robert Gaizauskas‡, Frank Schilder⋆, Mark Hepple‡, Graham Katz∗ and James Pustejovsky† † Brandeis University, {marc,jamesp}@cs.brandeis.edu ‡ University of Sheffield, {r.gaizauskas,m.hepple}@dcs.shef.ac.uk ⋆ Thomson Legal & Regulatory, frank.schilder@thomson.com, ∗ Stanford University, egkatz@stanford.edu Abstract The TempEval task proposes a simple way to evaluate automatic extraction of temporal relations. It avoids the pitfalls of evaluat- ing a graph of inter-related labels by defin- ing three sub tasks that allow pairwise eval- uation of temporal relations. The task not only allows straightforward evaluation, it also avoids the complexities of full tempo- ral parsing. 1 Introduction Newspaper texts, narratives and other texts describe events that occur in time and specify the temporal location and order of these events. Text comprehen- sion, amongst other capabilities, clearly requires the capability to identify the events described in a text and locate these in time. This capability is crucial to a wide range of NLP applications, from document summarization and question answering to machine translation. Recent work on the annotation of events and tem- poral relations has resulted in both a de-facto stan- dard for expressing these relations and a hand-built gold standard of annotated texts. TimeML (Puste- jovsky et al., 2003a) is an emerging ISO standard for annotation of events, temporal expressions and the anchoring and ordering relations between them. TimeBank (Pustejovsky et al., 2003b; Boguraev et al., forthcoming) was originally conceived of as a proof of concept that illustrates the TimeML lan- guage, but has since gone through several rounds of revisions and can now be considered a gold standard for temporal information. TimeML and TimeBank have already been used as the basis for automatic time, event and temporal relation annotation tasks in a number of research projects in recent years (Mani et al., 2006; Boguraev et al., forthcoming). An open evaluation challenge in the area of tem- poral annotation should serve to drive research for- ward, as it has in other areas of NLP. The auto- matic identification of all temporal referring expres- sions, events and temporal relations within a text is the ultimate aim of research in this area. However, addressing this aim in a first evaluation challenge was judged to be too difficult, both for organizers and participants, and a staged approach was deemed more effective. Thus we here present an initial eval- uation exercise based on three limited tasks that we believe are realistic both from the perspective of as- sembling resources for development and testing and from the perspective of developing systems capable of addressing the tasks. They are also tasks, which should they be performable automatically, have ap- plication potential. 2 Task Description The tasks as originally proposed were modified slightly during the course of resource development for the evaluation exercise due to constraints on data and annotator availability. In the following we de- scribe the tasks as they were ultimately realized in …

Content truncated to 500 words