Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3002–3008, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3002 Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games Subhajit Chaudhury IBM Research subhajit@jp.ibm.com Daiki Kimura IBM Research daiki@jp.ibm.com Kartik Talamadupula IBM Research krtalamad@us.ibm.com Michiaki Tatsubori IBM Research mich@jp.ibm.com Asim Munawar IBM Research asim@jp.ibm.com Ryuki Tachibana IBM Research ryuki@jp.ibm.com Abstract We show that Reinforcement Learn- ing (RL) methods for solving Text-Based Games (TBGs) often fail to generalize on un- seen games, especially in small data regimes. To address this issue, we propose Context Relevant Episodic State Truncation (CREST) for irrelevant token removal in observation text for improved generalization. Our method first trains a base model using Q-learning, which typically overfits the training games. The base model’s action token distribution is used to perform observation pruning that removes irrelevant tokens. A second boot- strapped model is then retrained on the pruned observation text. Our bootstrapped agent shows improved generalization in solving unseen TextWorld games, using 10x-20x fewer training games compared to previous state-of-the-art (SOTA) methods despite requiring fewer number of training episodes.1 1 Introduction Reinforcement Learning (RL) methods are increas- ingly being used for solving sequential decision- making problems from natural language inputs, like text-based games (Narasimhan et al., 2015; He et al., 2016; Yuan et al., 2018; Zahavy et al., 2018) chat-bots (Serban et al., 2017) and personal con- versation assistants (Dhingra et al., 2017; Li et al., 2017; Wu et al., 2016). In this work, we focus on Text-Based Games (TBGs), which require solving goals like “Obtain coin from the kitchen”, based on a natural language description of the agent’s observation of the environment. To interact with the environment, the agent issues text-based action commands (“go west”) upon which it receives a re- ward signal used for training the RL agent. TBGs serve as testbeds for interactive real-world tasks 1Our code is available at: www.github.com/IBM/context-relevant-pruning-textrl Goal: Who’s got a virtual machine and is about to play through an fast paced round of textworld? You do! Retrieve the coin in the balmy kitchen. Observation: You’ve entered a studio. You try to gain information on your surroundings by using a technique you call “looking.” You need an unguarded exit ? you should try going east. You need an unguarded exit? You should try go- ing south. You don’t like doors? Why not try going west, that entranceway is unblocked. Bootstrapped Policy Action: go south Figure 1: Our method retains context-relevant tokens from the observation text (shown in green) while prun- ing irrelevant tokens (shown in red). A second policy network re-trained on the pruned observations general- izes better by avoiding overfitting to unwanted tokens. like virtual-navigation agents on cellular phones at a shopping mall with user rating as the reward. Traditional text-based RL methods focus on the problems of partial observability and large action spaces. However, the topic of generalization to un- seen TBGs is less explored in the literature. We show that previous RL …

Content truncated to 500 words