A Dataset for Tracking Entities in Open Domain Procedural Text

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6408–6417, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6408 A Dataset for Tracking Entities in Open Domain Procedural Text Niket Tandon1 Keisuke Sakaguchi1 Bhavana Dalvi Mishra1 Dheeraj Rajagopal 2 Peter Clark1 Michal Guerquin1 Kyle Richardson1 Eduard Hovy2 1Allen Institute for Artificial Intelligence, Seattle, WA {nikett, keisukes, bhavanad, peterc, michalg, kyler}@allenai.org 2Department of Computer Science, CMU {dheeraj, hovy}@cs.cmu.edu Abstract We present the first dataset for tracking state changes in procedural text from arbitrary do- mains by using an unrestricted (open) vo- cabulary. For example, in a text describing fog removal using potatoes, a car window may transition between being foggy, sticky, opaque, and clear. Previous formulations of this task provide the text and entities involved, and ask how those entities change for just a small, pre-defined set of attributes (e.g., lo- cation), limiting their fidelity. Our solution is a new task formulation where given just a procedural text as input, the task is to gen- erate a set of state change tuples (entity, at- tribute, before-state, after-state) for each step, where the entity, attribute, and state values must be predicted from an open vocabulary. Using crowdsourcing, we create OPENPI1, a high-quality (91.5% coverage as judged by humans and completely vetted), and large- scale dataset comprising 29,928 state changes over 4,050 sentences from 810 procedural real- world paragraphs from WikiHow.com. A cur- rent state-of-the-art generation model on this task achieves 16.1% F1 based on BLEU met- ric, leaving enough room for novel model ar- chitectures. 1 Introduction By one estimate, only about 12% of what we under- stand from text is expressed explicitly (Graesser, 1981). This is especially apparent in text about ac- tions where the effects of actions are left unstated. Humans fill that gap easily with their common- sense but machines need to model these effects in the form of state changes. For example, when a potato is rubbed on a car window (to defog it), then the unstated effects of this action are the following state changes: windows becomes sticky, opaque, and the potato becomes dirty, etc. These changes can be tracked across the paragraph. An exem- plary use case of text with actions is procedural 1Download OPENPI at https://allenai.org/data/openpi Figure 1: Previous formulations of the state tracking task are restricted to a small, fixed set of pre-defined state change types that limits its fidelity to model real- world procedures (they cannot cover the blue part in this procedure comprising four steps). Our solution is a new task formulation to track an unrestricted (open) set of state changes (additionally covering blue). text (recipes, how-to guides, etc.) where modeling such state changes helps in various reasoning-based end tasks, e.g. automatic execution of biology ex- periments (Mysore et al., 2019), cooking recipes (Bollini et al., 2012) and everyday activities (Yang and Nyberg, 2015). While there has been great progress in tracking entity states in scientific processes (Dalvi et al., 2018), tracking ingredients in cooking recipes (Bosselut et …

Content truncated to 500 words