Language Understanding for Text-based Games using Deep Reinforcement Learning

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1–11, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Language Understanding for Text-based Games using Deep Reinforcement Learning Karthik Narasimhan∗ CSAIL, MIT karthikn@csail.mit.edu Tejas D Kulkarni∗ CSAIL, BCS, MIT tejask@mit.edu Regina Barzilay CSAIL, MIT regina@csail.mit.edu Abstract In this paper, we consider the task of learn- ing control policies for text-based games. In these games, all interactions in the vir- tual world are through text and the un- derlying state is not observed. The re- sulting language barrier makes such envi- ronments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This frame- work enables us to map text descriptions into vector representations that capture the semantics of the game states. We eval- uate our approach on two game worlds, comparing against baselines using bag-of- words and bag-of-bigrams for state rep- resentations. Our algorithm outperforms the baselines on both worlds demonstrat- ing the importance of learning expressive representations. 1 1 Introduction In this paper, we address the task of learning con- trol policies for text-based strategy games. These games, predecessors to modern graphical ones, still enjoy a large following worldwide.2 They of- ten involve complex worlds with rich interactions and elaborate textual descriptions of the underly- ing states (see Figure 1). Players read descriptions of the current world state and respond with natural language commands to take actions. Since the un- derlying state is not directly observable, the player has to understand the text in order to act, making it ∗Both authors contributed equally to this work. 1Code is available at http://people.csail.mit. edu/karthikn/mud-play. 2http://mudstats.com/ State 1: The old bridge You are standing very close to the bridge’s eastern foundation. If you go east you will be back on solid ground ... The bridge sways in the wind. Command: Go east State 2: Ruined gatehouse The old gatehouse is near collapse. Part of its northern wall has already fallen down ... East of the gatehouse leads out to a small open area surrounded by the remains of the castle. There is also a standing archway of- fering passage to a path along the old south- ern inner wall. Exits: Standing archway, castle corner, Bridge over the abyss Figure 1: Sample gameplay from a Fantasy World. The player with the quest of finding a secret tomb, is currently located on an old bridge. She then chooses an action to go east that brings her to a ruined gatehouse (State 2). challenging for existing AI programs to play these games (DePristo and Zubek, 2001). In designing an autonomous game player, we have considerable latitude when selecting an ad- equate state representation to use. The simplest method is to use a bag-of-words representation derived from the text description. However, this scheme disregards the ordering of words and the finer nuances of meaning that evolve from com- posing words into sentences and paragraphs. For instance, …

Content truncated to 500 words