Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 541–553 May 2-6, 2023 ©2023 Association for Computational Linguistics Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction Haoyu Wang1, Hongming Zhang2, Yuqian Deng1, Jacob R. Gardner1, Muhao Chen3 & Dan Roth1 1Department of Computer and Information Science, UPenn 2Tencent AI Lab, Seattle 3Department of Computer Science, USC {why16gzl, yuqiand, jacobrg, danroth}@seas.upenn.edu; hongmzhang@tencent.com; muhaoche@usc.edu Abstract In this paper, we seek to improve the faithful- ness of TEMPREL extraction models from two perspectives. The first perspective is to extract genuinely based on contextual description. To achieve this, we propose to conduct counter- factual analysis to attenuate the effects of two significant types of training biases: the event trigger bias and the frequent label bias. We also add tense information into event represen- tations to explicitly place an emphasis on the contextual description. The second perspec- tive is to provide proper uncertainty estimation and abstain from extraction when no relation is described in the text. By parameterization of Dirichlet Prior over the model-predicted cate- gorical distribution, we improve the model esti- mates of the correctness likelihood and make TEMPREL predictions more selective. We also employ temperature scaling to recalibrate the model confidence measure after bias mitigation. Through experimental analysis on MATRES, MATRES-DS, and TDDiscourse, we demon- strate that our model extracts TEMPREL and timelines more faithfully compared to SOTA methods, especially under distribution shifts. 1 Introduction Event temporal relation (TEMPREL) extraction is an essential step towards understanding narrative text, such as stories, novels, news, and guideline articles. With a robust temporal relation extrac- tor, one can easily construct a storyline from text and capture the trend of temporally connected event mentions. TEMPREL extraction is also broadly ben- eficial to various downstream tasks including clin- ical narrative processing (Jindal and Roth, 2013; Bethard et al., 2016), question answering (Llorens et al., 2015; Meng et al., 2017; Stricker, 2021), and schema induction (Chambers and Jurafsky, 2009; Wen et al., 2021; Li et al., 2021). Most existing TEMPREL extraction models are developed with data-driven machine learning ap- proaches, for which recent studies also incorporate A) I went to e1:SEE the doctor. However, I was more seriously e2:SICK. =⇒ e1 AFTER e2 B) Microsoft said it has e3:IDENTIFIED three companies for the China program to run through June. The company also e4:GIVES each partici- pating startup in the Seattle program $20,000 to create software. =⇒ e3 BEFORE e4 Figure 1: Examples of unfaithful extractions. BEFORE and AFTER that follow the arrows denote the extracted TEMPREL’s from the sentences by (Zhou et al., 2021). advanced learning and inference techniques such as structured prediction (Ning et al., 2017, 2018b; Han et al., 2019; Wang et al., 2020; Tan et al., 2021), graph representation (Mathur et al., 2021; Zhang et al., 2022), data augmentation (Ballesteros et al., 2020; Trong et al., 2022), and indirect supervision (Zhao et al., 2021; Zhou et al., 2021). These mod- els are prevalently built upon pretrained language …

Content truncated to 500 words