VLN-Trans: Translator for the Vision and Language Navigation Agent

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13219–13233 July 9-14, 2023 ©2023 Association for Computational Linguistics VLN-Trans: Translator for the Vision and Language Navigation Agent Yue Zhang Michigan State University zhan1624@msu.edu Parisa Kordjamshidi Michigan State University kordjams@msu.edu Abstract Language understanding is essential for the navigation agent to follow instructions. We ob- serve two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vi- sion abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applica- ble to multiple targets, thus not distinctive for selecting the target among the candidate view- points. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to- follow sub-instruction representations at each step. The translator needs to focus on the recog- nizable and distinctive landmarks based on the agent’s visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and de- sign specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks. 1 Introduction Vision-and-Language Navigation (VLN) (Ander- son et al., 2018) task requires an agent to under- stand and follow complex instructions to arrive at a destination in a photo-realistic simulated environ- ment. This cross-domain task attracts researchers from the communities of computer vision, natural language processing, and robotics (Gu et al., 2022; Wu et al., 2021; Francis et al., 2022). To solve the VLN task, one streamline of meth- ods is to build the connections between text and vision modalities by grounding the semantic infor- mation dynamically (Hong et al., 2020a; Qi et al., 2020a; An et al., 2021; Zhang and Kordjamshidi, 2022a). However, we observe two types of instruc- tions that make the grounding in the VLN task quite Enter the door and turn right passing the wall. Enter the shutters. Reach the entrance between the kitchen and the living room. Reach the entrance between the sofa and chairs. (a) (b) Instruction Candidate Views where which Figure 1: Two types of instructions that make the grounding in the VLN task challenging: (a) unrecogniz- able landmarks, (b) nondistinctive landmarks. In candi- date views, the largest image shows the target view. challenging. First, the instruction contains land- marks that are not recognizable by the navigation agent. For example, Figure 1(a), the agent can only see the “sofa”, “table” and “chair” in the target viewpoint, based on the learned vision representa- tions (He et al., 2016; Ren et al., 2015; Dosovitskiy et al., 2020). However, the instructor mentions land- marks of the “living room” and “kitchen” in the instruction, based on their prior knowledge about the environment, such as relating “sofa” to “living room”. Given the small size of the dataset designed for learning …

Content truncated to 500 words