What-if I ask you to explain: Explaining the effects of perturbations in procedural text

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3345–3355 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3345 What-if I ask you to explain: Explaining the effects of perturbations in procedural text Dheeraj Rajagopal1, Niket Tandon2, Bhavana Dalvi2, Peter Clark2, Eduard Hovy1 1Carnegie Mellon University 2Allen Institute for Artificial Intelligence {dheeraj,hovy}@cs.cmu.edu {niket,bhavanad,peterc}@allenai.org Abstract Our goal is to explain the effects of perturba- tions in procedural text, e.g., given a passage describing a rabbit’s life cycle, explain why ill- ness (the perturbation) may reduce the rabbit population (the effect). Although modern sys- tems are able to solve the original prediction task well (e.g., illness results in less rabbits), the explanation task - identifying the causal chain of events from perturbation to effect - re- mains largely unaddressed, and is the goal of this research. We present QUARTET, a system that constructs such explanations from para- graphs, by modeling the explanation task as a multitask learning problem. QUARTET con- structs explanations from the sentences in the procedural text, achieving ∼ 18 points better on explanation accuracy compared to several strong baselines on a recent process compre- hension benchmark. On an end task on this benchmark, we show a surprising finding that good explanations do not have to come at the expense of end task performance, in fact lead- ing to a 7% F1 improvement over SOTA. 1 Introduction Procedural text is common in natural language (in recipes, how-to guides, etc.) and finds many ap- plications such as automatic execution of biology experiments (Mysore et al., 2019), cooking recipes (Bollini et al., 2012) and everyday activities (Yang and Nyberg, 2015). However, the goal of proce- dural text understanding in these settings remains a major challenge and requires two key abilities, (i) understanding the dynamics of the world inside a procedure by tracking entities and what events happen as the narrative unfolds. (ii) understanding the dynamics of the world outside the procedure that can influence the procedure. While recent systems for procedural text compre- hension have focused on understanding the dynam- ics of the world inside the process, such as tracking Figure 1: Given a procedural text, the task is to explain the effect of the perturbation using the input sentences. entities and answering questions about what events happen, e.g., (Tandon et al., 2018; Bosselut et al., 2018; Henaff et al., 2017), the extent to which they understand the influences of outside events remains unclear. In particular, if a system fully understands a process, it should be able to predict what would happen if it was perturbed in some way due to an event from the outside world. Such counterfac- tual reasoning is particularly challenging because, rather than asking what happened (described in text), it asks about what would happen in an alter- native world where the change occurred. Recently, Tandon et al. (2019) introduced the WIQA dataset that contains such problems, re- quiring prediction of the effect of perturbations in a procedural text. They also presented several strong models …

Content truncated to 500 words