Causal Inference of Script Knowledge

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7583–7596, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7583 Causal Inference of Script Knowledge Noah Weber 1, Rachel Rudinger 2,3, Benjamin Van Durme 1 1Johns Hopkins University 2University of Maryland, College Park 3Allen Institute for AI Abstract When does a sequence of events define an ev- eryday scenario and how can this knowledge be induced from text? Prior works in induc- ing such scripts have relied on, in one form or another, measures of correlation between instances of events in a corpus. We argue from both a conceptual and practical sense that a purely correlation-based approach is insuf- ficient, and instead propose an approach to script induction based on the causal effect be- tween events, formally defined via interven- tions. Through both human and automatic evaluations, we show that the output of our method based on causal effects better matches the intuition of what a script represents. 1 Introduction Commonsense knowledge of everyday situations, as defined in terms of prototypical sequences of events, has long been held to play a major role in text comprehension and understanding (Minsky, 1974; Schank and Abelson, 1975, 1977; Bower et al., 1979; Abbott et al., 1985). Naturally, this has motivated a large body of work looking to learn such knowledge, such scripts,1 from text corpora through data-driven approaches. A minimal and often implicit requirement for any such approach is to resolve for any pair of events e1 and e2 what quantitative measure should be used to determine whether e2 should ”follow” e1 in script. That is, documents may serve as descrip- tions of events that occur in the same situation as other events: what function may we compute over the raw presence or absence of events in documents that is most useful for script induction? Chambers and Jurafsky (2008; 2009) adopted point-wise mutual information (PMI) (Church and 1For simplicity we will refer to these ‘prototypical event sequences’ as scripts throughout the paper, though it should be noted scripts as originally proposed contain further structure not captured in this definition. Figure 1: The events of Watching a sad movie, Eat- ing popcorn, and Crying, may highly co-occur in a hy- pothetical corpus. What distinguishes valid event pair inferences (event pairs linked in a commensense sce- nario; noted by checkmarks above) versus invalid infer- ences (noted by a ‘X’)? Hanks, 1990) between event mentions. Others em- ployed probabilities from a language model over event sequences (Jans et al., 2012; Rudinger et al., 2015; Pichotta and Mooney, 2016; Peng and Roth, 2016; Weber et al., 2018b), or other measures of event co-occurrence (Balasubramanian et al., 2013; Modi and Titov, 2014). In this work we ask: do measures rooted in co- occurrence best capture the notion of whether one event should follow another in a script? We posit that it does not, that while observed correlations between events indicate relatedness, relatedness is not the only factor in determining whether events form a meaningful script. …

Content truncated to 500 words