Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6764–6776 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning Subhajit Chaudhury, Sarathkrishna Swaminathan, Daiki Kimura, Prithviraj Sen, Keerthiram Murugesan, Rosario Uceda-Sosa, Michiaki Tatsubori, Achille Fokoue, Pavan Kapanipathi, Asim Munawar and Alexander Gray {subhajit, sarath.swaminathan, keerthiram.murugesan, asim, alexander.gray}@ibm.com {daiki, mich}@jp.ibm.com, {rosariou, achille, kapanipa}@us.ibm.com IBM Research Abstract Text-based reinforcement learning agents have predominantly been neural network-based mod- els with embeddings-based representation, learning uninterpretable policies that often do not generalize well to unseen games. On the other hand, neuro-symbolic methods, specifi- cally those that leverage an intermediate formal representation, are gaining significant attention in language understanding tasks. This is be- cause of their advantages ranging from inher- ent interpretability, the lesser requirement of training data, and being generalizable in scenar- ios with unseen data. Therefore, in this paper, we propose a modular, NEuro-Symbolic Tex- tual Agent (NESTA) that combines a generic semantic parser with a rule induction system to learn abstract interpretable rules as poli- cies. Our experiments on established text- based game benchmarks show that the pro- posed NESTA method outperforms deep rein- forcement learning-based techniques by achiev- ing better generalization to unseen test games and learning from fewer training interactions. 1 Introduction Text-based games (TBGs) (Côté et al., 2018) serve as popular sandbox environments for evaluat- ing natural language-based reinforcement learning. The agent observes the state of the game in pure text and issues a textual command to interact with the environment. TBGs are partially observable where the full state of the world is hidden and ac- tion commands facilitate the agent to explore the unobserved parts of the environment. The reward signal from the environment is used to improve the agent’s policy and make progress in the game. Text-based games sit at the intersection of two research areas, i.e., language understanding and reinforcement learning. Existing RL agents for TBGs primarily use embeddings for observation as representations and are fed to an action scorer for predicting the next action (Narasimhan et al., … You see an open pedal bin … There is a brown golf shoe on the floor and a blue moccasin in the cabinet. You are carrying a wet tissue. Observation Observation representation (Deep Neural network) Action scorer (Deep neural network) NESTA Deep RL Inductive Logic Programming (ILP) put(𝑥, 𝑦) :- carry 𝑥 ∧'atlocation 𝑥, 𝑦 State embedding (1) take brown golf shoes (2) put wet tissue into pedal bin (3) take blue moccasins from the cabinet (4) close pedal bin Admissible actions take blue moccasins from the cabinet put wet tissue into pedal bin Abstract Meaning Representation (AMR) Symbolic facts: carry(wet tissue) 0.2 0.4 0.91 0 0.2 0.4 0.6 0.8 1 TWCAgent BiKE+CBR (SOTA) Ours (NESTA) 50 45.91 31.8 0 10 20 30 40 50 60 TWCAgent BiKE+CBR (SOTA) Ours (NESTA) Norm score (higher is better) Steps (lower is better) carry-01 you tissue ARG0 ARG1 ARG1-of wet-01 Figure 1: Our …

Content truncated to 500 words