Neuro-Symbolic Reinforcement Learning with First-Order Logic

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3505–3511 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3505 Neuro-Symbolic Reinforcement Learning with First-Order Logic Daiki Kimura Masaki Ono Subhajit Chaudhury Ryosuke Kohita Akifumi Wachi Don Joven Agravante Michiaki Tatsubori Asim Munawar Alexander Gray IBM Research {daiki, moono, subhajit}@jp.ibm.com, {kohi, akifumi.wachi}@ibm.com don.joven.r.agravante@ibm.com, mich@jp.ibm.com, {asim, alexander.gray}@ibm.com Abstract Deep reinforcement learning (RL) methods of- ten require many trials before convergence, and no direct interpretability of trained poli- cies is provided. In order to achieve fast convergence and interpretability for the pol- icy in RL, we propose a novel RL method for text-based games with a recent neuro- symbolic framework called Logical Neural Network, which can learn symbolic and inter- pretable rules in their differentiable network. The method is first to extract first-order log- ical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly in- terpretable logical operators. Our experimen- tal results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark. 1 Introduction Deep reinforcement learning (RL) has been suc- cessfully applied to many applications, such as computer games, text-based games, and robot con- trol applications (Mnih et al., 2015; Narasimhan et al., 2015; Kimura, 2018; Yuan et al., 2018; Kimura et al., 2018). However, these methods re- quire many training trials for converging to the optimal action policy, and the trained action policy is not understandable for human operators. This is because, although the training results are sufficient, the policy is stored in a black-box deep neural net- work. These issues become critical problems when the human operator wants to solve a real-world problem and verify the trained rules. If the trained rules are understandable and modifiable, the human operator can control them and design an action re- striction. While using a symbolic (logical) format as representation for stored rules is suitable for achieving interpretability and quick training, it is difficult to train the logical rules with a traditional training approach. -= Studio =- I am required to announce that you are now in the studio. You don't like doors? Why not try going north, that entranceway is unblocked. You don't like doors? Why not try going south, that entranceway is unguarded. ∀𝑥 ∈ {east, west, south, north} Semantic parser AND OR ¬ Find 𝑥 Find 𝑥 ⋀ ⋁ ⋀ History Logical Neural Network 𝑜𝑏𝑠 𝑎𝑐𝑡𝑖𝑜𝑛 𝑟𝑒𝑤𝑎𝑟𝑑 Environment Action: “ Go 𝑥 ” ConceptNet … … FOL converter Visited 𝑥 ¬ Visited 𝑥 Figure 1: Overview of the proposed method. The agent takes a text observation from the environment, and the first-order logical facts are extracted from an FOL con- verter that uses a semantic parser, ConceptNet, and his- tory. The weights (shown by line thickness in this fig- ure) of the network are updated by these extracted pred- icate logics. Solid lines show one trained rule; when the agent finds a direction x and the direction x …

Content truncated to 500 words