Neuro-Symbolic Approaches for Text-Based Policy Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3073–3078 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3073 Neuro-Symbolic Approaches for Text-Based Policy Learning Subhajit Chaudhury, Prithviraj Sen, Masaki Ono, Daiki Kimura, Michiaki Tatsubori and Asim Munawar IBM Research subhajit@jp.ibm.com, senp@us.ibm.com, {moono, daiki, mich}@jp.ibm.com, asim@ibm.com Abstract Text-Based Games (TBGs) have emerged as important testbeds for reinforcement learning (RL) in the natural language domain. Previous methods using LSTM-based action policies are uninterpretable and often overfit the train- ing games showing poor performance to un- seen test games. We present SymboLic Action policy for Textual Environments (SLATE), that learns interpretable action policy rules from symbolic abstractions of textual obser- vations for improved generalization. We out- line a method for end-to-end differentiable symbolic rule learning and show that such symbolic policies outperform previous state- of-the-art methods in text-based RL for the coin collector environment from 5−10x fewer training games. Additionally, our method pro- vides human-understandable policy rules that can be readily verified for their logical consis- tency and can be easily debugged. 1 1 Introduction Text-based games are increasingly being used as a benchmark for progressing the state of the art in natural language RL. These games typically require solving goals that are defined by natural language descriptions such as “retrieve the coin in the cel- lar”. The agent receives a textual description of the scene and can interact with the environment using only textual commands such as “go north”, “take knife” upon which it receives a reward signal for completing the goal (or sub-goal). The action policy model is trained using such reward signals. Previous methods in text-based RL typically use memory-based recurrent systems for feature extrac- tion from textual observations (Narasimhan et al., 2015; Adolphs and Hofmann, 2020) or knowledge graph extraction (Ammanabrolu and Riedl, 2019; Ammanabrolu and Hausknecht, 2020) for better 1Reproducibility: We will release our code at https://github.com/subhajit1411/ slate-text-based-rl Observation: You find yourself in a bedroom. An usual one. I guess you better just go and list everything you see here. There is an exit to the north. Don't worry, it is unguarded. There is an exit to the south. Don't worry, it is unblocked. You don't like doors? Why not try going west, that entranceway is unguarded. Symbolic facts: hasExit(bedroom, north), hasExit(bedroom, south), hasExit(bedroom, west), hasVisited(bedroom, south) hasExit(x,y) hasVisited(x,y) hasCoin(x) ¬hasExit(x,y) ¬hasVisited(x,y) ¬hasCoin(x) ∧ ∧ ∧ Logic: go(x,y) Logic: take(x) go(x,y) ⃪ hasExit(x,y)^¬hasVisited(x,y)^¬hasCoin(x) take(x) ⃪ hasCoin(x) Final sampled output action: go west Possible actions: go(x,west), go(x,north) 0.5 0.5 0.99 0.83 ∧ 0.99 0.99 0.17 Figure 1: Overview of our neuro-symbolic rule learn- ing. SLATE learns interpretable action policy for each action verb, go and take, from first-order symbolic states. We show the raw weights for each AND gate (∧) which are thresholded to obtain the logical rules. state representation. However, Chaudhury et al. (2020) showed that previous methods in text-based RL do not generalize well to unseen test games. Furthermore, the learned policy in such cases is not interpretable and is difficult to …

Content truncated to 500 words