Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension

Proceedings of NAACL-HLT 2018, pages 1595–1604 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension Bhavana Dalvi Mishra1*, Lifu Huang2∗, Niket Tandon1, Wen-tau Yih1, Peter Clark1 1Allen Institute for AI, Seattle, 2Rensselaer Polytechnic Institute, Troy {bhavanad,nikett,scottyih,peterc}@allenai.org, {warrior.fu}@gmail.com Abstract We present a new dataset and models for com- prehending paragraphs about processes (e.g., photosynthesis), an important genre of text de- scribing a dynamic world. The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a chang- ing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints). The end-task, tracking the location and existence of entities through the text, is challenging because the causal effects of actions are often implicit and need to be inferred. We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input en- coding and span prediction. The new models improve accuracy by up to 19%. The dataset and models are available to the community at http://data.allenai.org/propara. 1 Introduction Building a reading comprehension (RC) system that is able to read a text document and to answer questions accordingly has been a long-standing goal in NLP and AI research. Impressive progress has been made in factoid-style reading compre- hension, e.g., (Seo et al., 2017a; Clark and Gard- ner, 2017), enabled by well-designed datasets and modern neural network models. However, these models still struggle with questions that require inference (Jia and Liang, 2017). Consider the paragraph in Figure 1 about pho- tosynthesis. While top systems on SQuAD (Ra- jpurkar et al., 2016) can reliably answer lookup questions such as: Q1: What do the roots absorb? (A: water, minerals) they struggle when answers are not explicit, e.g., Q2: Where is sugar produced? (A: in the leaf)1 ∗*Bhavana Dalvi Mishra and Lifu Huang contributed equally to this work. 1For example, the RC system BiDAF (Seo et al., 2017a) answers “glucose” to this question. Chloroplasts in the leaf of the plant trap light from the sun. The roots absorb water and min- erals from the soil. This combination of water and minerals flows from the stem into the leaf. Carbon dioxide enters the leaf. Light, water and minerals, and the carbon dioxide all com- bine into a mixture. This mixture forms sugar (glucose) which is what the plant eats. Q: Where is sugar produced? A: in the leaf Figure 1: A paragraph from ProPara about photosyn- thesis (bold added, to highlight question and answer el- ements). Processes are challenging because questions (e.g., the one shown here) often require inference about the process states. To answer Q2, it appears that a system needs knowl- edge of the world and the ability to reason with state transitions in multiple sentences: If carbon dioxide enters the leaf (stated), then …

Content truncated to 500 words