Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4018–4028 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4018 Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments Sonia Raychaudhuri1 Saim Wani2 Shivansh Patel2 Unnat Jain3 Angel X. Chang1 1Simon Fraser University 2IIT Kanpur 3UIUC 1{sraychau,angelx}@sfu.ca 2{saimdwani,shivanshpatel35}@gmail.com 3uj2@illinois.edu https://3dlg-hcvc.github.io/LAW-VLNCE Abstract In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions. A challenge in this task is how to handle ‘off the path’ scenarios where an agent veers from a reference path. Prior work supervises the agent with actions based on the shortest path from the agent’s location to the goal, but such goal-oriented supervision is often not in alignment with the instruction. Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow. In this work, we propose a simple and effective language-aligned supervision scheme, and a new metric that measures the number of sub-instructions the agent has completed during navigation. 1 Introduction Training agents to navigate in realistic environ- ments based on natural language instructions is a step towards building robots that understand hu- mans and can assist them in their daily chores. An- derson et al. (2018b) introduced the Vision-and- Language Navigation (VLN) task, where an agent navigates a 3D environment to follow natural lan- guage instructions. Much of the prior work on VLN assumes a discrete navigation graph (nav- graph), where the agent teleports between graph nodes, both in indoor (Anderson et al., 2018b) and outdoor (Chen et al., 2019; Mehta et al., 2020) set- tings. Krantz et al. (2020) reformulated the VLN task to a continuous environment (VLN-CE) by lifting the discrete paths to continuous trajectories, bringing the task closer to real-world scenarios. Krantz et al. (2020) supervised agent training with actions based on the shortest path from the agent’s location to the goal, following prior work in VLN (Fried et al., 2018; Tan et al., 2019; Hu et al., 2019; Anderson et al., 2019). However, as Jain et al. Go toward the white table on the patio Turn right and go inside through the sliding doors Stop next to the stove and island in the kitchen Go through the room with television and into the kitchen Goal-oriented path Language-aligned path Language-aligned waypoints Agent Location Towards goal location Towards language-aligned waypoint Supervision: 1 2 3 4 # 1 2 3 4 Goal Start Start Goal Figure 1: A language-aligned path (blue) in an in- struction following task may differ from the shortest path (red) to the goal. Language-aligned supervision (blue arrows) encourages the agent at any given loca- tion (dark circles) to move towards the nearest way- point on the language-aligned path and can hence be a better supervisory signal for instruction following than goal-oriented supervision (red arrows). (2019) observed, such supervision is goal-oriented and does not always correspond to following the natural language instruction. Our key idea is that language-aligned supervi- …

Content truncated to 500 words