EvEntS ReaLM: Event Reasoning of Entity States via Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1982 - 1997 December 7-11, 2022 ©2022 Association for Computational Linguistics EVENTS REALM: Event Reasoning of Entity States via Language Models Evangelia Spiliopoulou∗† Amazon AWS, AI Labs spilieva@amazon.com Artidoro Pagnoni∗ Univ. of Washington artidoro@cs. washington.edu Yonatan Bisk Carnegie Mellon University ybisk@cs.cmu.edu Eduard Hovy Carnegie Mellon University hovy@cs.cmu.edu Abstract This paper investigates models of event impli- cations. Specifically, how well models pre- dict entity state-changes, by targeting their un- derstanding of physical attributes. Nominally, Large Language models (LLM) have been ex- posed to procedural knowledge about how ob- jects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve per- formance of reported baseline results across multiple tasks. In particular, our results indi- cate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.1 1 Introduction Modeling the effect of actions on entities (event implications) is a fundamental problem in AI span- ning computer vision, cognitive science and natural language understanding. Most commonly referred to as the Frame Problem (McCarthy and Hayes, 1981), early solutions relied on a set of handcrafted rules and logical statements to model event impli- cations. However, such methods require substantial manual effort and fail to generalize. More recently, modeling event implications has reemerged under the guise of common sense reasoning within NLP (Sap et al., 2019b; Bisk et al., 2020b; Talmor et al., 2019) and action anticipation in Computer Vision (Damen et al., 2018; Bakhtin et al., 2019). Predicting event implications is a particularly difficult problem due to the complex nature of lan- guage and implicit knowledge required to answer such questions. For example, if we are given the * Equal contribution. † Work completed before joining AWS AI Labs. 1 https://github.com/spilioeve/eventsrealm Context: The robot holds a laptop. The robot forcefully throws the laptop. Context: Pick up the yogurt, bananas, and sorbet. Place the ingredients in a blender. Blend the mixture until it's smooth in texture. PiGLET Open PI What attributes changed: Laptop is broken, picked-up and its location is different. What attributes changed: 1. The cleanness, weight, volume and fullness of the blender changed. 2. The texture and appearance of the mixture changed. ● 14 attributes ● AI2 Thor Simulator ● 5k/2k/2k train/dev/test ● 51 in-domain attributes ● 40 out-of-domain attributes ● WikiHow articles ● 11k/1k/2k train/dev/test Entities: blender, mixture Entity: Laptop Figure 1: We use the PiGLET and OpenPI datasets to probe if LLMs contain the necessary grounded and world knowledge to reason about event implications. sentence the mug fell on the floor and we want to determine whether the mug is broken, we need to know of several facts such as the material of the mug, the fragility of ceramics, the hardness of the floor, etc. and also how to combine these facts to- …

Content truncated to 500 words