Deep Reinforcement Learning with a Natural Language Action Space

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1621–1630, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Deep Reinforcement Learning with a Natural Language Action Space Ji He∗, Jianshu Chen†, Xiaodong He†, Jianfeng Gao†, Lihong Li† Li Deng† and Mari Ostendorf∗ ∗Department of Electrical Engineering, University of Washington, Seattle, WA 98195, USA {jvking, ostendor}@uw.edu †Microsoft Research, Redmond, WA 98052, USA {jianshuc, xiaohe, jfgao, lihongli, deng}@microsoft.com Abstract This paper introduces a novel architec- ture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games. Termed a deep reinforcement relevance network (DRRN), the architecture repre- sents action and state spaces with sepa- rate embedding vectors, which are com- bined with an interaction function to ap- proximate the Q-function in reinforce- ment learning. We evaluate the DRRN on two popular text games, showing su- perior performance over other deep Q- learning architectures. Experiments with paraphrased action descriptions show that the model is extracting meaning rather than simply memorizing strings of text. 1 Introduction This work is concerned with learning strategies for sequential decision-making tasks, where a sys- tem takes actions at a particular state with the goal of maximizing a long-term reward. More specifi- cally, we consider tasks where both the states and the actions are characterized by natural language, such as in human-computer dialog systems, tutor- ing systems, or text-based games. In a text-based game, for example, the player (or system, in this case) is given a text string that describes the cur- rent state of the game and several text strings that describe possible actions one could take. After se- lecting one of the actions, the environment state is updated and revealed in a new textual description. A reward is given either at each transition or in the end. The objective is to understand, at each step, the state text and all the action texts to pick the most relevant action, navigating through the se- quence of texts so as to obtain the highest long- term reward. Here the notion of relevance is based on the joint state/action impact on the reward: an action text string is said to be “more relevant” (to a state text string) than the other action texts if taking that action would lead to a higher long- term reward. Because a player’s action changes the environment, reinforcement learning (Sutton and Barto, 1998) is appropriate for modeling long- term dependency in text games. There is a large body of work on reinforcement learning. Of most interest here are approaches leveraging neural networks because of their suc- cess in handling a large state space. Early work — TD-gammon — used a neural network to approxi- mate the state value function (Tesauro, 1995). Re- cently, inspired by advances in deep learning (Le- Cun et al., 2015; Hinton et al., 2012; Krizhevsky et al., 2012; Dahl et al., 2012), significant progress has been made by combining deep learning with reinforcement …

Content truncated to 500 words