Acquiring Structured Temporal Representation via Crowdsourcing: A Feasibility Study

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 178–185 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 178 Acquiring Structured Temporal Representation via Crowdsourcing: A Feasibility Study Yuchen Zhang Brandeis University yuchenz@brandeis.edu Nianwen Xue Brandeis University xuen@brandeis.edu Abstract Temporal Dependency Trees are a structured temporal representation that represents tem- poral relations among time expressions and events in a text as a dependency tree struc- ture. Compared to traditional pair-wise tem- poral relation representations, temporal de- pendency trees facilitate efficient annotations, higher inter-annotator agreement, and efficient computations. However, annotations on tem- poral dependency trees so far have only been done by expert annotators, which is costly and time-consuming. In this paper, we introduce a method to crowdsource temporal dependency tree annotations, and show that this representa- tion is intuitive and can be collected with high accuracy and agreement through crowdsourc- ing. We produce a corpus of temporal depen- dency trees, and present a baseline temporal dependency parser, trained and evaluated on this new corpus. 1 Introduction Temporal relation extraction is an important NLP task for a range of downstream applications, such as question answering, summarization, and sto- ryline generation. This task has attracted a sig- nificant amount of research interest (Pustejovsky et al., 2003a; Verhagen et al., 2007, 2010; Uz- Zaman et al., 2012; Bethard et al., 2016, 2017; Dligach et al., 2017; Leeuwenberg and Moens, 2017; Ning et al., 2017, 2018a,b; Zhang and Xue, 2018a,b). One practical challenge in temporal re- lation extraction is to represent the temporal rela- tions in a text in a way that is feasible for man- ual annotation and producing training data for ma- chine learning models. Given a text of n events and time expressions, there are �n 2 � possible re- lations if the temporal relation between all pairs of events and time expressions is annotated. This quickly becomes infeasible even for a text of mod- est length. One way to address this problem is to represent the temporal relations in a text as a Tem- poral Dependency Tree (TDT) structure (Zhang and Xue, 2018b). TDT models all time expres- sions and events in a text as “nodes” in a depen- dency tree, and temporal relations between each time/event and its parent time/event as “edges” in the tree. Figure 1 gives an example text and its TDT. Each (parent, child) pair in Figure 1 is an- notated with a temporal relation. The number of temporal relations that need to be annotated in a text is therefore linear to the number of events and time expressions in a text, making the annotation task feasible. At the same time, additional tem- poral relations can be inferred as needed based on the TDT structure. For example, in Figure 1 since “1918” includes the “born” event and “1929” in- cludes the “won” event, it can be inferred that the “born” event occurred before the “won” event. By providing annotators with detailed guide- lines and training them in multiple iterations, Zhang and …

Content truncated to 500 words