WIQA: A dataset for “What if...” reasoning over procedural text

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6076–6085, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6076 WIQA: A dataset for "What if..." reasoning over procedural text Niket Tandon*, Bhavana Dalvi Mishra∗, Keisuke Sakaguchi, Antoine Bosselut, Peter Clark Allen Institute for Artificial Intelligence, Seattle, WA {nikett,bhavanad,keisukes,antoineb,peterc}@allenai.org Abstract We introduce WIQA, the first large-scale dataset of "What if..." questions over proce- dural text. WIQA contains three parts: a col- lection of paragraphs each describing a pro- cess, e.g., beach erosion; a set of crowdsourced influenct graphs for each paragraph, describ- ing how one change affects another; and a large (40k) collection of "What if...?" multiple- choice questions derived from the graphs. For example, given a paragraph about beach ero- sion, would stormy weather result in more or less erosion (or have no effect)? The task is to answer the questions, given their asso- ciated paragraph. WIQA contains three kinds of questions: perturbations to steps mentioned in the paragraph; external (out-of-paragraph) perturbations requiring commonsense knowl- edge; and irrelevant (no effect) perturbations. We find that state-of-the-art models achieve 73.8% accuracy, well below the human perfor- mance of 96.3%. We analyze the challenges, in particular tracking chains of influences, and present the dataset as an open challenge to the community. 1 Introduction Procedural text is common in language, but chal- lenging to comprehend because it describes a dy- namically changing world. While recent systems for procedural text comprehension can answer questions about what events happen, e.g., (Bosselut et al., 2018; Henaff et al., 2017; Dalvi et al., 2018), the extent to which they understand the influences between those events remains unclear. One important test of understanding is to pre- dict what would happen if a process was perturbed in some way, requiring understanding and tracing ∗*Niket Tandon and Bhavana Dalvi Mishra contributed equally to this work. Procedural Text (simplified): Erosion by the ocean: 1. Wind creates waves in the ocean. 2. The waves wash onto the beaches. 3. The waves hit rocks on the beach. 4. Tiny parts of the rock break off. 5. The rocks become smaller. Figure 1: WIQA contains procedural paragraphs, crowdsourced influence graphs associated with them, and a large collection of “Does changeX result in changeY?” (what-if) questions, derived from the graphs. the chain of influences through a paragraph. How- ever, to date there is no dataset available to help develop this capability. We aim to fill this gap with WIQA1, the first large-scale dataset testing "What if..." reasoning over procedural text. WIQA contains 40.7K questions, for 379 pro- cess paragraphs. To efficiently create the questions, crowdworkers created 2107 influence graphs (IGs) for the paragraphs, describing how one perturbation positively or negatively influences another (Fig- 1The dataset is available at http://data.allenai.org/wiqa/ 6077 ure 1). Questions were then derived from paths in the graphs, each asking how the change described in one node affects another. Each question is a tem- plated, multiple …

Content truncated to 500 words