LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5745–5754 October 12–17, 2022. 5745 LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation Yue Zhang Michigan State University zhan1624@msu.edu Parisa Kordjamshidi Michigan State University kordjams@msu.edu Abstract Understanding spatial and visual information is essential for a navigation agent who fol- lows natural language instructions. The cur- rent Transformer-based VLN agents entangle the orientation and vision information, which limits the gain from the learning of each in- formation source. In this paper, we design a neural agent with explicit Orientation and Vi- sion modules. Those modules learn to ground spatial information and landmark mentions in the instructions to the visual environment more effectively. To strengthen the spatial reasoning and visual perception of the agent, we design specific pre-training tasks to feed and better utilize the corresponding modules in our final navigation model. We evaluate our approach on both Room2room (R2R) and Room4room (R4R) datasets and achieve the state of the art results on both benchmarks. 1 Introduction Vision and Language Navigation (VLN) prob- lem (Anderson et al., 2018) has attracted increasing attention from the communities of computer vision, natural language processing, and robotics because of its broad real-world applications. In this prob- lem setting, the goal of a navigation agent is to move to a target location in a photo-realistic simu- lated environment by following a detailed natural language instruction, e.g., “Walk into the bedroom. Walk past the bedroom door. Wait at the laundry room door.”. Two kinds of simulators are used to create the dataset and the corresponding prob- lem formulation: discrete trajectories (Anderson et al., 2018) and continuous navigation trajecto- ries (Krantz et al., 2020). In this paper, we work on the discrete one, that an agent traverses a pre- defined connectivity graph by selecting the adjacent viewpoint with a higher probability of correspond- ing to the instruction at each step. The earlier research in the VLN area can be di- vided into two categories. The first category of left, right, 90 degree left turn, forward … go to table, walk to sofa, … candidate viewpoint1 candidate viewpoint2 candidate viewpoint3 Spatial reasoning ability Vision perception ability Pass through the blue chair on your left, and then go upstairs. Instruction Figure 1: Spatial reasoning helps leveraging orientation clues, such as left and 90 degree; visual perception ability grounds mentioned landmarks, such as table, sofa, and chair. With these two abilities, the agent selects the candidate viewpoint corresponding to the instruction at each navigation step. The green arrow shows the ground-truth viewpoint. models mostly depends on the encoder-decoder framework for encoding the text and visual in- formation, establishing the connections between them with the attention mechanism, and decod- ing the actions (Anderson et al., 2018; Ma et al., 2019; Fried et al., 2018). The second category of works learns to model the semantic structure which implicitly or explicitly enhances the textual- visual matching (Hong et al., 2020a,b; Qi et al., 2020; Zhang et al., 2021; Li et al., 2021; …

Content truncated to 500 words