Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4412–4417 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4412 Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions Peter A. Jansen School of Information, University of Arizona, Tucson, AZ pajansen@email.arizona.edu Abstract The recently proposed ALFRED challenge task aims for a virtual robotic agent to com- plete complex multi-step everyday tasks in a virtual home environment from high-level nat- ural language directives, such as “put a hot piece of bread on a plate”. Currently, the best-performing models are able to complete less than 5% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language direc- tives into detailed multi-step sequences of ac- tions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to generate gold multi-step plans from language directives alone without any vi- sual input in 26% of unseen cases. When a small amount of visual information is incorpo- rated, namely the starting location in the vir- tual environment, our best-performing GPT-2 model successfully generates gold command sequences in 58% of cases. Our results sug- gest that contextualized language models may provide strong visual semantic planning mod- ules for grounded virtual agents. 1 Introduction Simulated virtual environments with steadily in- creasing fidelity are allowing virtual agents to learn to perform high-level tasks that couple language un- derstanding, visual planning, and embodied reason- ing through sensorimotor grounded representations (Gordon et al., 2018; Puig et al., 2018; Wijmans et al., 2019). The ALFRED challenge task recently proposed by Shridhar et al. (2020) requires a virtual robotic agent to complete everyday tasks (such as “put cold apple slices on the table”) in one of 120 in- teractive virtual home environments by generating and executing complex visually-grounded seman- tic plans that involve movable objects, irreversible state changes, and an egocentric viewpoint. Inte- grating natural language task directives with one 1 2 3 4 5 6 {goto, countertop} {pick up, fork} {goto, sink basin} {clean, fork} {goto, drawer} {put, fork, drawer} “Wash the fork and put it away” Directive Figure 1: An example of the ALFRED grounded language task. In this work, we focus on visual semantic planning – from the textual directive alone (top), our model predicts a visual semantic plan of {command, argument} tuples (cap- tions) that matches the gold plan without requiring visual input (images). of the most complex interactive virtual agent envi- ronments to date is challenging, with the current best performing systems successfully completing less than 5% of ALFRED tasks in unseen environ- ments1, while common baseline models generally complete less than 1% of tasks successfully. In this work we explore the visual semantic plan- ning task in ALFRED, where the high-level natu- ral language task directive is converted into a de- tailed sequence of actions in the AI2-THOR 2.0 virtual environment (Kolve et al., 2017) that will accomplish that goal (see Figure 1). In contrast to previous approaches to …

Content truncated to 500 words