ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5367–5380 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5367 ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning Rujun Han1 Xiang Ren1 Nanyun Peng1,2 1University of Southern California 2University of California, Los Angeles {rujunhan,xiangren}@usc.edu;violetpeng@cs.ucla.edu Abstract While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that re- quire event temporal reasoning, which is essen- tial for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discrim- inate sentences from their corrupted counter- parts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event tem- poral reasoning. This Effective CONtinual pre-training framework for Event Temporal reasoning (ECONET) improves the PTLMs’ fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art perfor- mances in most of our downstream tasks.1 1 Introduction Reasoning event temporal relations is crucial for natural language understanding, and facilitates many real-world applications, such as tracking biomedical histories (Sun et al., 2013; Bethard et al., 2015, 2016, 2017), generating stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020), and forecasting social events (Li et al., 2020; Jin et al., 2020). In this work, we study two prominent event temporal reasoning tasks as shown in Figure 1: event relation extraction (ERE) (Chambers et al., 2014; Ning et al., 2018; O’Gorman et al., 2016; Mostafazadeh et al., 2016) that predicts temporal relations between a pair of events, and machine reading comprehension (MRC) (Ning et al., 2020; Zhou et al., 2019) where a passage and a question 1Reproduction code, training data and models are available here: https://github.com/PlusLabNLP/ECONET. Figure 1: Top: an example illustrating the difference between ERE and QA / MRC samples of event tempo- ral reasoning. Bottom: our targeted masking strategy for ECONET v.s. random masking in PTLMs. about event temporal relations is presented, and models need to provide correct answers using the information in a given passage. Recent approaches leveraging large pre-trained language models (PTLMs) achieved state-of-the- art results on a range of event temporal reasoning tasks (Ning et al., 2020; Pereira et al., 2020; Wang et al., 2020; Zhou et al., 2020c; Han et al., 2019b). Despite the progress, vanilla PTLMs do not focus on capturing event temporal knowledge that can be used to infer event relations. For example, in Figure 1, an annotator of the QA sample can easily infer from the temporal indicator “following” that “transfer” happens BEFORE “preparing the paper- work”, but a fine-tuned RoBERTa model predicts that “transfer” has no such relation with the event “preparing the paperwork.” Plenty of such cases ex- ist in our error analysis on PTLMs for event tempo- ral relation-related tasks. We hypothesize that such deficiency is …

Content truncated to 500 words