Be Consistent! Improving Procedural Text Comprehension using Label Consistency

Proceedings of NAACL-HLT 2019, pages 2347–2356 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2347 Be Consistent! Improving Procedural Text Comprehension using Label Consistency Xinya Du1∗ Bhavana Dalvi Mishra2 Niket Tandon2 Antoine Bosselut2 Wen-tau Yih2 Peter Clark2 Claire Cardie1 1Department of Computer Science, Cornell University, Ithaca, NY {xdu, cardie}@cs.cornell.edu 2Allen Institute for Artificial Intelligence, Seattle, WA {bhavanad, nikett, antoineb, scottyih, peterc}@allenai.org Abstract Our goal is procedural text comprehension, namely tracking how the properties of entities (e.g., their location) change with time given a procedural text (e.g., a paragraph about pho- tosynthesis, a recipe). This task is challeng- ing as the world is changing throughout the text, and despite recent advances, current sys- tems still struggle with this task. Our approach is to leverage the fact that, for many proce- dural texts, multiple independent descriptions are readily available, and that predictions from them should be consistent (label consistency). We present a new learning framework that leverages label consistency during training, al- lowing consistency bias to be built into the model. Evaluation on a standard benchmark dataset for procedural text, ProPara (Dalvi et al., 2018), shows that our approach signif- icantly improves prediction performance (F1) over prior state-of-the-art systems. 1 Introduction We address the task of procedural text comprehen- sion, namely tracking how the properties of enti- ties (e.g., their location) change with time through- out the procedure (e.g., photosynthesis, a cook- ing recipe). This ability is an important part of text understanding, allowing the reader to infer un- stated facts such as how ingredients change during a recipe, what the inputs and outputs of a scientific process are, or who met whom in a news article about a political meeting. Although several proce- dural text comprehension systems have emerged recently (e.g., EntNet (Henaff et al., 2017), NPN (Bosselut et al., 2018), and ProStruct (Tandon et al., 2018)), they still make numerous prediction errors. A major challenge is that fully annotated training data for this task is expensive to collect, because *Work done while at the Allen Institute for Artificial In- telligence. (1) ...oxygen is given off... (2) ...the plant produces oxygen... (3) ...is used to create sugar and oxygen... Figure 1: Fragments from three independent texts about photosynthesis. Although (1) is ambiguous as to whether oxygen is being created or merely moved, evidence from (2) and (3) suggests it is being created, helping to correctly interpret (1). More generally, en- couraging consistency between predictions from differ- ent paragraphs about the same process/procedure can improve performance. many state changes by multiple entities may occur in a single text, requiring complex annotation. To address this challenge, and thus improve per- formance, our goals are two-fold: first, to better leverage the training data for procedural text com- prehension that is available, and second, to uti- lize additional unlabeled data for the task (semi- supervised learning). Our approach in each case is to exploit label consistency, the property that two distinct texts covering the same procedure should be generally consistent …

Content truncated to 500 words