Modeling Preconditions in Text with a Crowd-sourced Dataset

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3818–3828 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3818 Modeling Preconditions in Text with a Crowd-sourced Dataset Heeyoung Kwon1, Mahnaz Koupaee1, Pratyush Singh1, Gargi Sawhney1, Anmol Shukla1, Keerthi Kumar Kallur1, Nathanael Chambers2 and Niranjan Balasubramanian1 1 Stony Brook University, Stony Brook, New York 2 US Naval Academy, Annapolis, MD {heekwon, mkoupaee, niranjan}@cs.stonybrook.edu nchamber@usna.edu Abstract Preconditions provide a form of logical con- nection between events that explains why some events occur together and information that is complementary to the more widely stud- ied relations such as causation, temporal order- ing, entailment, and discourse relations. Mod- eling preconditions in text has been hampered in part due to the lack of large scale labeled data grounded in text. This paper introduces PeKo, a crowd-sourced annotation of precon- ditions between event pairs in newswire, an or- der of magnitude larger than prior text annota- tions. To complement this new corpus, we also introduce two challenge tasks aimed at mod- eling preconditions: (i) Precondition Identifi- cation – a standard classification task defined over pairs of event mentions, and (ii) Precon- dition Generation – a generative task aimed at testing a more general ability to reason about a given event. Evaluation on both tasks shows that modeling preconditions is challeng- ing even for today’s large language models (LM). This suggests that precondition knowl- edge is not easily accessible in LM-derived representations alone. Our generation results show that fine-tuning an LM on PeKo yields better conditional relations than when trained on raw text or temporally-ordered corpora. 1 Introduction Recognizing logical connections between events in text is important for comprehensive document understanding and to improve global coherence in language generation systems. There is a rich body of work in identifying relations between textual events which covers causation (Mirza et al., 2014), temporal relations (Pustejovsky et al., 2003), tex- tual entailment (Dagan et al., 2005), and discourse relations (Blair-Goldensohn and McKeown, 2006). In this work, we focus on the precondition re- lation, which offers a general view of why certain events occur together in the world. This is not eas- ily deduced from other event-event relations. Tem- poral ordering systems can sequence the order in which events occurred (Bethard, 2013; Chambers et al., 2014; Han et al., 2019) but can’t explain why they occurred at all. Which events in a sequence were by chance, and which were required? Tex- tual entailment identifies event paraphrases (Berant et al., 2015) and some causation (Girju, 2003a), but their view misses the broader look at enabling events like preconditions. Let the following serve as an example: I heard a bird sing above as I turned the key in the door. It opened with a push. You can sequence these four events in order, but an ordering does not understand the why of the situation. One of these events (sing) is clearly not relevant to the door opening. How do we know that turning the key is a precondition to opened …

Content truncated to 500 words