Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1004–1015 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Mapping Instructions and Visual Observations to Actions with Reinforcement Learning Dipendra Misra†, John Langford‡, and Yoav Artzi† † Dept. of Computer Science and Cornell Tech, Cornell University, New York, NY 10044 {dkm, yoav}@cs.cornell.edu ‡ Microsoft Research, New York, NY 10011 jcl@microsoft.com Abstract We propose to directly map raw visual ob- servations and text input to actions for in- struction execution. While existing ap- proaches assume access to structured envi- ronment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about lin- guistic and visual input. We use reinforce- ment learning in a contextual bandit set- ting to train a neural network agent. To guide the agent’s exploration, we use re- ward shaping with different forms of su- pervision. Our approach does not re- quire intermediate representations, plan- ning procedures, or training different mod- els. We evaluate in a simulated environ- ment, and show significant improvements over supervised learning and common re- inforcement learning variants. 1 Introduction An agent executing natural language instructions requires robust understanding of language and its environment. Existing approaches addressing this problem assume structured environment represen- tations (e.g.,. Chen and Mooney, 2011; Mei et al., 2016), or combine separately trained models (e.g., Matuszek et al., 2010; Tellex et al., 2011), includ- ing for language understanding and visual reason- ing. We propose to directly map text and raw im- age input to actions with a single learned model. This approach offers multiple benefits, such as not requiring intermediate representations, plan- ning procedures, or training multiple models. Figure 1 illustrates the problem in the Blocks environment (Bisk et al., 2016). The agent ob- serves the environment as an RGB image using a camera sensor. Given the RGB input, the agent North South East West Put the Toyota block in the same row as the SRI block, in the first open space to the right of the SRI block Move Toyota to the immediate right of SRI, evenly aligned and slightly separated Move the Toyota block around the pile and place it just to the right of the SRI block Place Toyota block just to the right of The SRI Block Toyota, right side of SRI Figure 1: Instructions in the Blocks environment. The instructions all describe the same task. Given the ob- served RGB image of the start state (large image), our goal is to execute such instructions. In this task, the direct-line path to the target position is blocked, and the agent must plan and move the Toyota block around. The small image marks the target and an example path, which includes 34 steps. must recognize the blocks and their layout. To un- derstand the instruction, the agent must identify the block to move (Toyota block) and the destina- tion (just right of the SRI block). This requires solving semantic and grounding problems. For example, consider …

Content truncated to 500 words