SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 1–9, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 1: TEMPEVAL-3: Evaluating Time Expressions, Events, and Temporal Relations Naushad UzZaman♣1, Hector Llorens♦1, Leon Derczynski♥, Marc Verhagen♠, James Allen♣ and James Pustejovsky♠ ♣: University of Rochester, USA; ♦: University of Alicante, Spain ♥: Department of Computer Science, University of Sheffield, UK ♠: Computer Science Department, Brandeis University, USA 1: Nuance Communications naushad@cs.rochester.edu, hllorens@dlsi.ua.es, leon@dcs.shef.ac.uk Abstract Within the SemEval-2013 evaluation exercise, the TempEval-3 shared task aims to advance research on temporal information processing. It follows on from TempEval-1 and -2, with: a three-part struc- ture covering temporal expression, event, and tem- poral relation extraction; a larger dataset; and new single measures to rank systems – in each task and in general. In this paper, we describe the participants’ approaches, results, and the observations from the results, which may guide future research in this area. 1 Introduction The TempEval task (Verhagen et al., 2009) was added as a new task in SemEval-2007. The ultimate aim of research in this area is the automatic identification of temporal ex- pressions (timexes), events, and temporal relations within a text as specified in TimeML annotation (Pustejovsky et al., 2005). However, since addressing this aim in a first evaluation challenge was deemed too difficult a staged approach was suggested. TempEval (henceforth TempEval-1) was an initial evaluation exercise focusing only on the categorization of temporal relations and only in English. It included three relation types: event-timex, event-dct,1 and relations be- tween main events in consecutive sentences. TempEval-2 (Verhagen et al., 2010) extended TempEval-1, growing into a multilingual task, and con- sisting of six subtasks rather than three. This included event and timex extraction, as well as the three relation tasks from TempEval-1, with the addition of a relation task where one event subordinates another. TempEval-3 (UzZaman et al., 2012b) is a follow-up to TempEval 1 and 2, covering English and Spanish. TempEval-3 is different from its predecessors in a few respects: 1DCT stands for document creation time Size of the corpus: the dataset used has about 600K word silver standard data and about 100K word gold stan- dard data for training, compared to around 50K word cor- pus used in TempEval 1 and 2. Temporal annotation is a time-consuming task for humans, which has limited the size of annotated data in previous TempEval exer- cises. Current systems, however, are performing close to the inter-annotator reliability, which suggests that larger corpora could be built from automatically annotated data with minor human reviews. We want to explore whether there is value in adding a large automatically created sil- ver standard to a hand-crafted gold standard. End-to-end temporal relation processing task: the temporal relation classification tasks are performed from raw text, i.e. participants need to extract their own events and temporal expressions first, determine which ones to link and then obtain the relation types. In previous Tem- pEvals, …

Content truncated to 500 words