Sub-Instruction Aware Vision-and-Language Navigation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3360–3376, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3360 Sub-Instruction Aware Vision-and-Language Navigation Yicong Hong∗1, Cristian Rodriguez-Opazo∗1, Qi Wu2, Stephen Gould1 1The Australian National University, 2University of Adelaide 1,2Australian Centre for Robotic Vision {yicong.hong, cristian.rodriguez, stephen.gould}@anu.edu.au qi.wu01@adelaide.edu.au Abstract Vision-and-language navigation requires an agent to navigate through a real 3D envi- ronment following natural language instruc- tions. Despite significant advances, few pre- vious works are able to fully utilize the strong correspondence between the visual and textual sequences. Meanwhile, due to the lack of inter- mediate supervision, the agent’s performance at following each part of the instruction cannot be assessed during navigation. In this work, we focus on the granularity of the visual and language sequences as well as the traceability of agents through the completion of an instruc- tion. We provide agents with fine-grained an- notations during training and find that they are able to follow the instruction better and have a higher chance of reaching the target at test time. We enrich the benchmark dataset Room- to-Room (R2R) with sub-instructions and their corresponding paths. To make use of this data, we propose effective sub-instruction attention and shifting modules that select and attend to a single sub-instruction at each time-step. We implement our sub-instruction modules in four state-of-the-art agents, compare with their baseline models, and show that our proposed method improves the performance of all four agents. We release the Fine-Grained R2R dataset (FGR2R) and the code at https://github. com/YicongHong/Fine-Grained-R2R. 1 Introduction Creating an agent that can navigate through an unknown environment following natural language instructions has been a dream of human-beings for many years. Such an agent needs to possess the ability to perceive its environment, understand the instructions and learn the relationship between * Authors contributed equally Figure 1: Visual navigation with sub-instruction and sub-path pairs. We enrich the R2R dataset by pro- viding fine-grained matching between sub-instructions and viewpoints along the ground-truth path. these two streams of information. Recently, An- derson et al. (2018b) proposed the vision-and- language navigation (VLN) task that formalized such requirements through an evaluation of an agent’s ability to follow natural language instruc- tions in photo-realistic environments. Despite the significant progress made by recent approaches, there is little evidence that agents learn the correspondence between observations and in- structions. Hu et al. (2019) found that a modified self-monitoring agent (Ma et al., 2019a), could achieve similar performance with (success rate 40.5%) and without (success rate 39.7%) visual information. Among other reasons, such as dataset bias, the result suggests that this agent gains little from having the two streams of information. We argue that one of the main reasons be- 3361 hind this is that current methods are not ade- quately teaching agents the relationship between perception—things that the robot is observing— and parts of the instructions. Since datasets do not provide such information agents can only use the ground-truth trajectory as a learning signal. More- over, given the lack …

Content truncated to 500 words