Exploring the Capacity of Pretrained Language Models for Reasoning about Actions and Change

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4629–4643 July 9-14, 2023 ©2023 Association for Computational Linguistics Exploring the Capacity of Pretrained Language Models for Reasoning about Actions and Change Weinan He1, Canming Huang1, Zhanhao Xiao2∗, Yongmei Liu1∗ 1Dept. of Computer Science, Sun Yat-sen University, Guangzhou 510006, China 2School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou 510665, China {heweinan, huangcm}@mail2.sysu.edu.cn, xiaozhanhao@gpnu.edu.cn, ymliu@mail.sysu.edu.cn Abstract Reasoning about actions and change (RAC) is essential to understand and interact with the ever-changing environment. Previous AI re- search has shown the importance of fundamen- tal and indispensable knowledge of actions, i.e., preconditions and effects. However, traditional methods rely on logical formalization which hinders practical applications. With recent transformer-based language models (LMs), rea- soning over text is desirable and seemingly fea- sible, leading to the question of whether LMs can effectively and efficiently learn to solve RAC problems. We propose four essential RAC tasks as a comprehensive textual bench- mark and generate problems in a way that min- imizes the influence of other linguistic require- ments (e.g., grounding) to focus on RAC. The resulting benchmark, TRAC, encompassing problems of various complexities, facilitates a more granular evaluation of LMs, precisely targeting the structural generalization ability much needed for RAC. Experiments with three high-performing transformers indicate that ad- ditional efforts are needed to tackle challenges raised by TRAC. 1 Introduction Reasoning about actions and change (RAC) has been a central issue since the early days in AI (Mc- Carthy, 1963) and received much attention from the NLP community (Li et al., 2021; Zellers et al., 2021; Dalvi et al., 2018). Classical AI has long recognized the importance of actions and charac- terized the fundamental knowledge about actions: preconditions and effects (Reiter, 2001). Precondi- tions are the conditions that must be satisfied when actions are executed, while effects define the result. Consider the example where an agent is moving containers in a dock: Suppose a red container is on top of a green one, and the agent is given an instruction to “move the green container to another ∗Corresponding author platform”. To achieve such a goal, it needs to know the preconditions of moving a container (there is not any other container on top of it) and the effect of such actions. While traditional approaches provide sound and effective reasoning by capturing preconditions and effects, they rely on expensive and difficult for- malization. Consequently, inducing knowledge from text and reasoning directly is becoming more preferable. As transformers have shown their promising potential in many linguistic tasks (Liu et al., 2019; Raffel et al., 2020), we set out to probe whether pretrained language models can master the ability of reasoning about actions and changes. Previous approaches involving actions in NLP are more application-centric and usually contain specific tasks (e.g. instruction following, predic- tion) (Li et al., 2021; Zellers et al., 2021) with- out considering both preconditions and effects. In general these tasks are not verified logically. On the other hand, …

Content truncated to 500 words