Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning

Proceedings of NAACL-HLT 2019, pages 3557–3565 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3557 Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning Prithviraj Ammanabrolu School of Interactive Computing Georgia Institute of Technology Atlanta, GA raj.ammanabrolu@gatech.edu Mark O. Riedl School of Interactive Computing Georgia Institute of Technology Atlanta, GA riedl@cc.gatech.edu Abstract Text-based adventure games provide a plat- form on which to explore reinforcement learning in the context of a combinatorial action space, such as natural language. We present a deep reinforcement learning architecture that represents the game state as a knowledge graph which is learned during exploration. This graph is used to prune the action space, enabling more efficient exploration. The question of which action to take can be reduced to a question-answering task, a form of transfer learning that pre-trains certain parts of our architecture. In experi- ments using the TextWorld framework, we show that our proposed technique can learn a control policy faster than baseline alterna- tives. We have also open-sourced our code at https://github.com/rajammanabrolu/KG- DQN. 1 Introduction Natural language communication can be used to affect change in the real world. Text adventure games, in which players must make sense of the world through text descriptions and declare ac- tions through natural language, can provide a step- ping stone toward more real-world environments where agents must communicate to understand the state of the world and indirectly affect change in the world. Text adventure games are also useful for developing and testing reinforcement learning algorithms that must deal with the partial observ- ability of the world (Narasimhan et al., 2015; He et al., 2016). In text adventure games, the agent receives an incomplete textual description of the current state of the world. From this information, and pre- vious interactions with the world, a player must determine the next best action to take to achieve some quest or goal. The player must then com- pose a textual description of the action they in- tend to make and receive textual feedback of the effects of the action. Formally, a text-based game is a partially observable Markov decision process (POMDP), represented as a 7-tuple of ⟨S, T, A, Ω, O, R, γ⟩ representing the set of en- vironment states, conditional transition probabil- ities between states, words used to compose text commands, observations, observation conditional probabilities, reward function, and the discount factor respectively (Cˆot´e et al., 2018). In text-based games, the agent never has access to the true underlying world state and has to rea- son about how to act in the world based only on the textual observations. Additionally, the agent’s ac- tions must be expressed through natural language commands, ensuring that the action space is com- binatorially large. Thus, text-based games pose a different set of challenges than traditional video games. Text-based games require a greater un- derstanding of previous context to be able to ex- plore the state-action space more effectively. Such games have historically proven to be difficult to play for AI agents, and …

Content truncated to 500 words