Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1041–1050 June 6–11, 2021. ©2021 Association for Computational Linguistics 1041 Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information Jialu Li Hao Tan UNC Chapel HillMohit Bansal {jialuli, airsplay, mbansal}@cs.unc.edu Abstract Vision language navigation is the task that re- quires an agent to navigate through a 3D en- vironment based on natural language instruc- tions. One key challenge in this task is to ground instructions with the current visual in- formation that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction re- quired for the next action. However, differ- ent words have different functions in a sen- tence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like de- pendencies and phrase structures can aid the agent to locate important parts of the instruc- tion. Hence, in this paper, we propose a naviga- tion agent that utilizes syntax information de- rived from a dependency tree to enhance align- ment between the instruction and the current visual scenes. Empirically, our agent outper- forms the baseline model that does not use syn- tax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the cur- rent visual information via qualitative visual- izations.1 1 Introduction Vision-Language Navigation defines the task of requiring an agent to navigate through a visual environment based on natural language instruc- tions (Anderson et al., 2018b; Misra et al., 2018; Chen et al., 2019; Jain et al., 2019; Nguyen and Daumé III, 2019; Thomason et al., 2020). This task poses several challenges. To complete this task, an embodied agent needs to perceive the surround- ing environment, understand the given natural lan- guage instructions, and most importantly, ground 1Code and models: https://github.com/ jialuli-luka/SyntaxVLN Navigation Steps: Parse Tree: Instruction: Walk forward then turn right at the stairs then go down the stairs. walk forward then turn at stairs go stairs then right the down . the Figure 1: An example in the Room-to-Room task. We generate the dependency parse tree for the instruction. The words are grouped by the head node (highlighted in the tree). Each sub-instruction (i.e., grouped words) corresponds to one step in the navigation with the same color. Modifiers in red boxes can be easily identified from the tree structure. (or align) the instruction in the visual scenes. In this paper, we aim to make one step towards grounding natural language instructions with visual environ- ment via syntax-enriched alignment. Recently, several approaches were proposed to solve the Vision-Language Navigation task with better interactions between natural language in- structions and visual scenes (Fried et al., 2018; Wang et al., 2019; Landi et al., 2019; Wang et al., 2020a; Huang et al., 2019a; Hu et al., 2019; Ma- …

Content truncated to 500 words