A Comprehensive Evaluation and Correction of the TimeBank Corpus

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2919–2927 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2919 A Comprehensive Evaluation and Correction of the TimeBank Corpus Mustafa Ocal, Antonela Radas, Jared Hummer, Karine Megerdoomian*, & Mark A. Finlayson Florida International University Knight Foundation School of Computing and Information Sciences CASE Building, Room 362, 11200 S.W. 8th Street, Miami, FL USA 33199 MITRE Corporation, 7515 Colshire Dr, McLean, VA USA 22102* {mocal001, arada002, jhumm001, markaf}@fiu.edu karine@mitre.org* Abstract TimeML is an annotation scheme for capturing temporal information in text. The developers of TimeML built the TimeBank corpus to both validate the scheme and provide a rich dataset of events, temporal expressions, and temporal relationships for training and testing temporal analysis systems. In our own work we have been developing methods aimed at TimeML graphs for detecting (and eventually automatically correcting) temporal inconsistencies, extracting timelines, and assessing temporal indeterminacy. In the course of this investigation we identified numerous previously unrecognized issues in the TimeBank corpus, including multiple violations of TimeML annotation guide rules, incorrectly disconnected temporal graphs, as well as inconsistent, redundant, missing, or otherwise incorrect annotations. We describe our methods for detecting and correcting these problems, which include: (a) automatic guideline checking (109 violations); (b) automatic inconsistency checking (65 inconsistent files); (c) automatic disconnectivity checking (625 incorrect breakpoints); and (d) manual comparison with the output of state-of-the-art automatic annotators to identify missing annotations (317 events, 52 temporal expressions). We provide our code as well as a set of patch files that can be applied to the TimeBank corpus to produce a corrected version for use by other researchers in the field1. Keywords: TimeML, TimeBank, Annotation Evaluation, Automatic Annotation Correction 1. Introduction TimeML is a temporal annotation scheme for capturing events, temporal expressions, and temporal relations in natural language texts (Pustejovsky et al., 2003a). The TimeBank corpus was released as a reference cor- pus for TimeML which provides extensive annotated data for training and testing temporal analysis systems that produce TimeML (Pustejovsky et al., 2003b). The TimeBank corpus has been used in much Natural Lan- guage Processing (NLP) research, including works on event detection (UzZaman and Allen, 2010; F¨arber and Rettinger, 2015; Bansal et al., 2018; Veyseh et al., 2021), temporal expression recognition (Kolomiyets and Moens, 2009; Zhong et al., 2017; Chen et al., 2019), and temporal link extraction (Mani et al., 2006; Mir- roshandel and Ghassem-Sani, 2011; Kadir et al., 2016; Ning et al., 2018). If there are errors in TimeBank, they could potentially affect the accuracy of all works that use it. In prior work we have been using TimeBank as a start- ing point for developing methods for detecting temporal inconsistency, measuring temporal indeterminacy, and automatically correcting and enriching temporal graphs. In the course of that work, we have identified a num- 1Data and code may be downloaded from https:// doi.org/10.34703/gzx1-9v95/EFNL6H. Approved for Public Release; Distribution Unlimited. Public Release Case Number 22-0288. ©2022 The MITRE Corporation and Florida International …

Content truncated to 500 words