Causal Reasoning of Entities and Events in Procedural Texts

Findings of the Association for Computational Linguistics: EACL 2023, pages 415–431 May 2-6, 2023 ©2023 Association for Computational Linguistics Causal Reasoning About Entities and Events in Procedural Texts Li Zhang♣∗, Hainiu Xu♣∗, Yue Yang♣, Shuyan Zhou♢, Weiqiu You♣, Manni Arora♣, Chris Callison-Burch♣ ♣University of Pennsylvania ♢Carnegie Mellon University {zharry,seacow,yueyang1,weiqiuy,manni,ccb}@seas.upenn.edu {shuyanzh}@cs.cmu.edu Abstract Entities and events are crucial to natural lan- guage reasoning and common in procedural texts. Existing work has focused either exclu- sively on entity state tracking (e.g., whether a pan is hot) or on event reasoning (e.g., whether one would burn themselves by touching the pan), while these two tasks are often causally related. We propose CREPE, the first bench- mark on causal reasoning of event plausibil- ity and entity states. We show that most lan- guage models, including GPT-3, perform close to chance at .35 F1, lagging far behind hu- man at .87 F1. We boost model performance to .59 F1 by creatively representing events as programming languages while prompting lan- guage models pretrained on code. By injecting the causal relations between entities and events as intermediate reasoning steps in our repre- sentation, we further boost the performance to .67 F1. Our findings indicate not only the chal- lenge that CREPE brings for language models, but also the efficacy of code-like prompting combined with chain-of-thought prompting for multihop event reasoning.1 1 Introduction Event-centric natural language processing (Chen et al., 2021b) is one of the leading paradigms in machine understanding of texts. This line of work focuses on first extracting entities and events from texts (Yang et al., 2019; Du and Cardie, 2020) and then making inferences about them (Li et al., 2020; Du et al., 2021). Even with the recent advances of large language models (LLMs), reasoning about events remains challenging as it requires highly contextual information and ample common-sense knowledge. For example, the event “adding water to a pan containing hot oil” causes the event “there is a sizzling sound” to happen, while “heat up an ∗Equal contribution. 1Data and code can be found at https://github.com/z harry29/causal_reasoning_of_entities_and_events. Figure 1: Example of our task CREPE. A procedure including a goal and some steps are provided. A model needs to predict the change in the likelihood of an event throughout the procedure. We show that predicting en- tity states as an intermediate step improves performance. empty pan” does not. Any model that can draw the correct conclusion given these contexts is expected to have access to some implicit knowledge about these entities and events. One type of text which demonstrates these chal- lenges is procedural text, namely sequences of events, such as how-to instructions, recipes, nat- ural processes, scientific protocols, etc. Procedural texts describe an environment that changes dynam- ically through a sequence of steps. Therefore, the exact environment configuration is often implicit. In the previous cooking example, whether “there is a sizzling sound” depends on what steps have taken place. With these interesting challenges coupled with the added benefit of application to robotics (Brohan et al., 2022) and …

Content truncated to 500 words