Keep CALM and Explore: Language Models for Action Generation in Text-based Games

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8736–8754, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8736 Keep CALM and Explore: Language Models for Action Generation in Text-based Games Shunyu Yao†, Rohan Rao†, Matthew Hausknecht‡, Karthik Narasimhan† †Princeton University ‡ Microsoft Research {shunyuy, rohanr, karthikn}@princeton.edu matthew.hausknecht@microsoft.com Abstract Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. In this paper, we propose the Contextual Ac- tion Language Model (CALM) to generate a compact set of action candidates at each game state. Our key insight is to train language mod- els on human gameplay, where people demon- strate linguistic priors and a general game sense for promising actions conditioned on game history. We combine CALM with a re- inforcement learning agent which re-ranks the generated action candidates to maximize in- game rewards. We evaluate our approach us- ing the Jericho benchmark (Hausknecht et al., 2019a), on games unseen by CALM during training. Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model. Surprisingly, on half of these games, CALM is competitive with or better than other models that have ac- cess to ground truth admissible actions.∗ 1 Introduction Text-based games have proven to be useful testbeds for developing agents that operate in language. As interactions in these games (input observations, ac- tion commands) are through text, they require solid language understanding for successful gameplay. While several reinforcement learning (RL) models have been proposed recently (Narasimhan et al., 2015; He et al., 2015; Hausknecht et al., 2019a; Ammanabrolu and Riedl, 2019), combinatorially large action spaces continue to make these games challenging for these approaches. The action space problem is exacerbated by the fact that only a tiny fraction of action commands are admissible in any given game state. An admis- sible action is one that is parseable by the game ∗Code and data are available at https://github. com/princeton-nlp/calm-textgame. Observation: You are in the living room. There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. You are carrying: A brass lantern ... Random Actions: close door, north a, eat troll with egg, . . . CALM (n-gram) Actions: enter room, leave room, lock room, open door, close door, knock on door, . . . CALM (GPT-2) Actions: east, open case, get rug, turn on lantern, move rug, unlock case with key, ... Next Observation: With a great effort, the rug is moved to one side of the room, revealing the dusty cover of a closed trap door... Figure 1: Sample gameplay from ZORK1 along with action sets generated by two variants of CALM. The game recognizes a vocabulary size of 697, resulting in more than 6974 ≈ 200 billion potential 4-word actions. ‘move rug’ is the optimal action to take here and is gen- …

Content truncated to 500 words