Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5412–5417, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5412 Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events Miguel Ballesteros Rishita Anubhai Shuai Wang Nima Pourdamghani Yogarshi Vyas Jie Ma Parminder Bhatia Kathleen McKeown∗ Yaser Al-Onaizan Amazon AI {ballemig,ranubhai,wshui,nimpourd,yogarshi,jieman,parmib,mckeownk,onaizan}@amazon.com Abstract In this paper, we propose a neural architec- ture and a set of training methods for order- ing events by predicting temporal relations. Our proposed models receive a pair of events within a span of text as input and they iden- tify temporal relations (Before, After, Equal, Vague) between them. Given that a key chal- lenge with this task is the scarcity of annotated data, our models rely on either pretrained rep- resentations (i.e. RoBERTa, BERT or ELMo), transfer and multi-task learning (by leverag- ing complementary datasets), and self-training techniques. Experiments on the MATRES dataset of English documents establish a new state-of-the-art on this task. 1 Introduction The task of temporal ordering of events involves predicting the temporal relation between a pair of input events in a span of text (Figure 1). This task is challenging as it requires deep understanding of temporal aspects of language and the amount of annotated data is scarce. Albright (e1, came) to the State Department to (e2, offer) condolences. Figure 1: Example from the MATRES dataset. The relation between (e1, came) and (e2, offer) is Before. Note that for the same span there may be other relation pairs. The MATRES dataset (Ning et al., 2018) has become a de facto standard for temporal order- ing of events.1 It contains 13,577 pairs of events annotated with a temporal relation (Before, After, Equal, Vague) within 256 English documents (and ∗Kathleen McKeown is an Amazon Scholar and a Pro- fessor at Columbia University. 1https://github.com/qiangning/MATRES 20 more for evaluation) from TimeBank2 (Puste- jovsky et al., 2003), AQUAINT3 (Graff, 2002) and Platinum (UzZaman et al., 2013). In this paper, we present a set of neural archi- tectures for temporal ordering of events. Our main model (Section 2) is similar to the temporal order- ing models designed by Goyal and Durrett (2019), Liu et al. (2019a) and Ning et al. (2019). Our main contributions are: (1) a neural archi- tecture that can flexibly adapt different encoders and pretrained word embedders to form a contex- tual pairwise argument representation. Given the scarcity of training data, (2) we explore the ap- plication of an existing framework for Scheduled Multitask-Learning (henceforth SMTL) (Kiper- wasser and Ballesteros, 2018) by leveraging com- plementary (temporal and non temporal) informa- tion to our models; this imitates pretraining and finetuning. This consumes timex information in a different way than Goyal and Durrett (2019). (3) A self-training method that incorporates the pre- dictions of our model and learns from them; we test it jointly with the SMTL method. Our baseline model that uses RoBERTa (Liu et al., 2019b) already surpasses the state-of-the-art by 2 F1 points. Applying SMTL techniques af- …

Content truncated to 500 words