Systems’ Agreements and Disagreements in Temporal Processing: An Extensive Error Analysis of the TempEval-3 Task

Agreements and Disagreements in Temporal Processing: An Extensive Error Analysis of the TempEval-3 Systems Tommaso Caselli†, Roser Morante‡ †CLCG – University of Groningen, ‡CLTL Lab – VU Amsterdam The Netherlands †t.caselli@gmail.com|rug.nl, ‡r.morantevallejo@vu.nl Abstract In this article we review Temporal Processing systems that participated in the TempEval-3 task as a basis to develop our own system, that we also present and release. The system incorporates high level lexical semantic features, obtaining the best scores for event detection (F1-Class 72.24) and second best result for temporal relation classification from raw text (F1 29.69) when evaluated on the TempEval-3 data. Additionally, we analyse the errors of all TempEval-3 systems for which the output is publicly available with the purpose of finding out what are the weaknesses of current approaches. Although incorporating lexical semantics features increases the performance of our system, the error analysis shows that systems should incorporate inference mechanisms and world knowledge, as well as having strategies to compensate for data skewness. Keywords: temporal processing, error analysis, written corpora 1. Introduction Any discourse, spoken or written, contains temporally con- nected linguistic mentions, such as events and temporal ex- pressions (timexes). Relations between these mentions can be meaningfully interpreted by using models of time, which allow to connect events on a timeline (temporal anchoring) and to understand complex sequences of events (temporal ordering). Temporal relations (TRs) provide a model and a set of properties to account for the connections between pairs of entities. Temporal Processing (TP) is a task consisting in automat- ically identifying and classifying basic entities and their relations, such as event-event (e-e), and event-timex (e- t). Temporally aware Natural Language Processing (NLP) systems are crucial not only to generate timelines and sto- rylines (Vossen et al., 2015), but also in decision sup- port systems, summarization and textual entailment appli- cations, question answering systems, and document archiv- ing, among others. Since the release of the TimeBank cor- pus (Pustejovsky et al., 2003) there has been a renewed in- terest in the area of TP, which has resulted in the celebra- tion of several evaluation campaigns1 and in the creation of corpora and tools in languages other than English. 2 The TempEval-3 campaign (UzZaman et al., 2013) is the lat- est campaign on open-domain TP in English. The major contribution of TempEval-3 is the release of a platform for the development and evaluation of end-to-end TP systems based on the TimeML mark-up language (Pustejovsky et al., 2003). After the TempEval-3 evaluation a new dataset has been re- leased, the TimeBank-Dense corpus (Cassidy et al., 2014). This corpus has been developed to address one of the ma- jor shortcomings of the TempEval-3 dataset, namely lack 1TempEval (Verhagen et al., 2007; Verhagen et al., 2010; Uz- Zaman et al., 2013), Clinical TempEval (Bethard et al., 2016; Bethard et al., 2017), Q-A TempEval (Llorens et al., 2015). 2For an extended list of available TimeBanks see (Caselli and Sprugnoli, 2017). of connectivity between all possible e-e and e-t pairs, by completing the transitive closure. The outcome …

Content truncated to 500 words