Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7519 - 7532 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas Raphael Schumann Computational Linguistics Heidelberg University, Germany Stefan Riezler Computational Linguistics & IWR Heidelberg University, Germany {rschuman|riezler}@cl.uni-heidelberg.de Abstract Vision and language navigation (VLN) is a challenging visually-grounded language un- derstanding task. Given a natural language navigation instruction, a visual agent interacts with a graph-based environment equipped with panorama images and tries to follow the de- scribed route. Most prior work has been con- ducted in indoor scenarios where best results were obtained for navigation on routes that are similar to the training routes, with sharp drops in performance when testing on unseen environ- ments. We focus on VLN in outdoor scenarios and find that in contrast to indoor VLN, most of the gain in outdoor VLN on unseen data is due to features like junction type embedding or heading delta that are specific to the respective environment graph, while image information plays a very minor role in generalizing VLN to unseen outdoor areas. These findings show a bias to specifics of graph representations of ur- ban environments, demanding that VLN tasks grow in scale and diversity of geographical en- vironments.1 1 Introduction Vision and language navigation (VLN) is a chal- lenging task that requires the agent to process nat- ural language instructions and ground them in a visual environment. The agent is embodied in the environment and receives navigation instructions. Based on the instructions, the observed surround- ings, and the current trajectory the agent decides its next action. Executing this action changes the position and/or heading of the agent within the en- vironment, and eventually the agent follows the described route and stops at the desired goal loca- tion. The most common evaluation metric in VLN is the proportion of successful agent navigations, called task completion (TC). 1Code: map2seq_vlnhttps://github.com/raphael-sch/ Data & Demo: https://map2seq.schumann.pub/ vln/ While early work on grounded navigation was confined to grid-world scenarios (MacMahon et al., 2006; Chen and Mooney, 2011), recent work has studied VLN in outdoor environment consisting of real-world urban street layouts and correspond- ing panorama pictures (Chen et al., 2019). Recent agent models for outdoor VLN treat the task as a sequence-to-sequence problem where the instruc- tions text is the input and the output is a sequence of actions (Chen et al., 2019; Xiang et al., 2020; Zhu et al., 2021b). In contrast to indoor VLN (An- derson et al., 2018; Ku et al., 2020), these works only consider a seen scenario, i.e., the agent is tested on routes that are located in the same area as the training routes. However, studies of indoor VLN (Zhang et al., 2020) show a significant per- formance drop when testing in previously unseen areas. The main goal of our work is to study outdoor VLN in unseen areas, pursuing the research ques- tion of which representations of an …

Content truncated to 500 words