ULN: Towards Underspecified Vision-and-Language Navigation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6394 - 6412 December 7-11, 2022 ©2022 Association for Computational Linguistics ULN: Towards Underspecified Vision-and-Language Navigation Weixi Feng Tsu-Jui Fu Yujie Lu William Yang Wang UC Santa Barbara {weixifeng, tsu-juifu, yujielu, william}@cs.ucsb.edu Abstract Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions. Despite the significant performance improve- ment, the wide use of fine-grained instructions fails to characterize more practical linguistic variations in reality. To fill in this gap, we in- troduce a new setting, namely Underspecified vision-and-Language Navigation (ULN), and associated evaluation datasets. ULN evaluates agents using multi-level underspecified instruc- tions instead of purely fine-grained or coarse- grained, which is a more realistic and general setting. As a primary step toward ULN, we propose a VLN framework that consists of a classification module, a navigation agent, and an Exploitation-to-Exploration (E2E) module. Specifically, we propose to learn Granularity Specific Sub-networks (GSS) for the agent to ground multi-level instructions with minimal additional parameters. Then, our E2E module estimates grounding uncertainty and conducts multi-step lookahead exploration to improve the success rate further. Experimental results show that existing VLN models are still brittle to multi-level language underspecification. Our framework is more robust and outperforms the baselines on ULN by „10% relative success rate across all levels. 1 1 Introduction Vision-and-Language Navigation (VLN) allows a human user to command or instruct an embodied agent to reach target locations using verbal instruc- tions. For this application to step out of curated datasets in real-world settings, the agents must gen- eralize to many linguistic variations of human in- structions. Despite significant progress in VLN datasets (Anderson et al., 2018b; Chen et al., 2019; Ku et al., 2020; Shridhar et al., 2020) and agent design (Fried et al., 2018; Li et al., 2021; Min et al., 1Our code and data are available at https://github. com/weixi-feng/ULN. Figure 1: Navigation results of a baseline (left) and our VLN framework (right) with multi-level underspecified instructions (L0-L3). Trajectories are curved for demon- stration. Note that the baseline stops early and fails to reach the target position with L1-L3. Our agent man- ages to reach the goal across all levels. 2021), it remains a question whether existing mod- els are generalized and robust enough to deal with all kinds of language variations. For the language input in an indoor environment, some datasets focus on long and detailed instruc- tions with the route description at every step to achieve fine-grained language grounding (Ander- son et al., 2018b; Ku et al., 2020) or long-horizon navigation (Jain et al., 2019; Zhu et al., 2020a). For instance, from Room-to-Room (R2R) (Anderson et al., 2018b), to Room-Across-Room (RxR) (Ku et al., 2020), the average instruction length in- creases from 29 to 129 words. Other datasets have coarse-grained instructions like REVERIE (Qi et al., 2020) or SOON (Zhu et al., 2021a). Agents are trained and evaluated on a single granu- 6394 larity or one type of …

Content truncated to 500 words