Holistic Evaluation of Automatic TimeML Annotators

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1444–1453 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1444 Holistic Evaluation of Automatic TimeML Annotators Mustafa Ocal, Adrian Perez, Antonela Radas, & Mark A. Finlayson Knight Foundation School of Computing and Information Sciences Florida International University CASE Building, Room 362, 11200 S.W. 8th Street, Miami, FL USA 33199 {mocal001, apere946, arada002, markaf}@fiu.edu Abstract TimeML is a scheme for representing temporal information (times, events, & temporal relations) in texts. Although automatic TimeML annotation is challenging, there has been notable progress, with F1s of 0.8–0.9 for events and time detection subtasks, and F1s of 0.5–0.7 for relation extraction. Individually, these subtask results are reasonable, even good, but when combined to generate a full TimeML graph, is overall performance still acceptable? We present a novel suite of eight metrics, combined with a new graph-transformation experimental design, for holistic evaluation of TimeML graphs. We apply these metrics to four automatic TimeML annotation systems (CAEVO, TARSQI, CATENA, and CLEARTK). We show that on average 1/3 of the TimeML graphs produced using these systems are inconsistent, and there is on average 1/5 more temporal indeterminacy than the gold-standard. We also show that the automatically generated graphs are on average 109 edits from the gold-standard, which is 1/3 toward complete replacement. Finally, we show that the relationship individual subtask performance and graph quality is non-linear: small errors in TimeML subtasks result in rapid degradation of final graph quality. These results suggest current automatic TimeML annotators are far from optimal and significant further improvement would be useful. Keywords: TimeML, Timelines, Evaluation Metrics 1. Introduction TimeML is an annotation scheme for representing tem- poral information in natural language texts. Automatic TimeML annotation is a challenging task, not only be- cause it comprises three subtasks (time expression de- tection, event detection, and TimeML relation extrac- tion), but also because temporal information is often implicit in the commonsense meaning of the language. There are several systems that can automatically gener- ate TimeML graphs by combining the output of compo- nents that each focus on one of the individual subtasks. When evaluated in isolation, the components perform well on event detection (0.80–0.85 F1 score) and time expression detection (0.82–0.93 F1 score), while au- tomatic relation extraction trails both in performance (0.50–0.63 F1) and coverage (4–11 out of 25 relation types extracted). Despite these respectable individual performances, the question remains of how the perfor- mance of the individual components relates to overall performance, and how that affects a logical next task, namely, timeline extraction. Limited coverage of rela- tion types is especially problematic, because while a reported F1 for relation extraction might be respectable, even good, all extant systems only extract a subset of relation types, and how these omissions effect later tasks has not been evaluated. Furthermore, running each com- ponent in isolation and then combining the results does not attend to question of consistency, which is critical to the utility of the …

Content truncated to 500 words