Annotating Temporal Dependency Graphs via Crowdsourcing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5368–5380, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5368 Annotating Temporal Dependency Graphs via Crowdsourcing Jiarui Yao1, Haoling Qiu2, Bonan Min2, and Nianwen Xue1 1Brandeis University {jryao, xuen}@brandeis.edu 2Raytheon BBN Technologies {haoling.qiu, bonan.min}@raytheon.com Abstract We present the construction of a corpus of 500 Wikinews articles annotated with temporal de- pendency graphs (TDGs) that can be used to train systems to understand temporal relations in text. We argue that temporal dependency graphs, built on previous research on narra- tive times and temporal anaphora, provide a representation scheme that achieves a good balance between completeness and practical- ity in temporal annotation. We also provide a crowdsourcing strategy to annotate TDGs, and demonstrate the feasibility of this approach with an evaluation of the quality of the anno- tation, and the utility of the resulting data set by training a machine learning model on this data set. This data set is publicly available1. 1 Introduction Understanding temporal relations between events in a text is an important part of understand- ing the “meaning” of text. With the wide adoption of machine learning methods in natural language processing, the ability to achieve a large-scale high- quality temporally annotated data set has been the bottleneck in advancing the state of the art in this area. Even though the first temporal annotation scheme, TimeML (Pustejovsky et al., 2003a; Saur´ı et al., 2006), was proposed over a decade ago, tem- porally annotated data is still relatively scarce. The largest data set that we are aware of is the data set used in TempEval-3 (UzZaman et al., 2013), and it consists of 276 articles from the TimeBank Corpus (Pustejovsky et al., 2003b) and the AQUAINT Cor- pus. This data set was later re-annotated by (Ning et al., 2018) to improve its annotation consistency using a crowdsourcing approach. There are many challenges that have contributed to this state of affairs. Temporal relations are often 1https://github.com/Jryao/temporal_ dependency_graphs_crowdsourcing confounded with modalities (How do you order an event that actually happened with one that might happen?). Some events are ambiguous between an instantaneous and stative reading (Does “marriage” refer to the start of the marriage or does it refer to the state when marriage is in effect?) and this complicates its temporal relation with other events. While these have all contributed to the difficulty in temporal annotation, the main challenge is a practi- cal one. The general assumption in temporal anno- tation has been that the temporal relation between every pair of events in a text has to be specified in order to fully understand the temporal relations in a text. This amounts to constructing a fully con- nected graph in which every event is connected to another event. With this pair-wise approach, a text of n events has �n 2 � event pairs that need to be con- sidered. As the value of n increases, the number of event pairs quickly becomes very large. In prac- tice, attempts to …

Content truncated to 500 words