Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4837 - 4851 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration Xiwen Liang1, Fengda Zhu2, Lingling Li3, Hang Xu4, Xiaodan Liang1† 1Shenzhen Campus of Sun Yat-sen University, Shenzhen 2Monash University 3Sun Yat-sen University 4Huawei Noah’s Ark Lab Abstract Vision-language navigation (VLN) is a chal- lenging task due to its large searching space in the environment. To address this prob- lem, previous works have proposed some methods of fine-tuning a large model that pretrained on large-scale datasets. How- ever, the conventional fine-tuning methods require extra human-labeled navigation data and lack self-exploration capabilities in en- vironments, which hinders their generaliza- tion of unseen scenes. To improve the ability of fast cross-domain adaptation, we propose Prompt-based Environmental Self-exploration (ProbES), which can self-explore the envi- ronments by sampling trajectories and auto- matically generates structured instructions via a large-scale cross-modal pretrained model (CLIP). Our method fully utilizes the knowl- edge learned from CLIP to build an in-domain dataset by self-exploration without human la- beling. Unlike the conventional approach of fine-tuning, we introduce prompt-based learn- ing to achieve fast adaptation for language embeddings, which substantially improves the learning efficiency by leveraging prior knowl- edge. By automatically synthesizing trajectory- instruction pairs in any environment without human supervision and efficient prompt-based learning, our model can adapt to diverse vision- language navigation tasks, including VLN and REVERIE. Both qualitative and quantitative results show that our ProbES significantly im- proves the generalization ability of the naviga- tion model*. 1 Introduction Teaching a robot to navigate following a natural language instruction has a broad impact in the field of human-robotic interaction. Many related tasks have been proposed to delve into this problem. The †Corresponding author. *Code will be released at https://github.com/ liangcici/Probes-VLN. vision-language navigation (VLN) task (Anderson et al., 2018) is proposed where an agent is required to navigate in a photo-realistic environment step- by-step following a natural language instruction. Recent tasks (Qi et al., 2020; Zhu et al., 2021) focus on target objects localization that asks an agent to identify an object in an unseen room. Solving these tasks requires an agent to obtain a vision-text alignment ability that locates related objects and executes corrective actions according to the instruction. However, collecting a large-scale VLN dataset is difficult and laborious since anno- tating the semantic of a trajectory within a sentence costs times of labor than annotating an image. Ex- isting navigation datasets are relatively small-scale, and learning on such datasets hinders the agent to obtain a good generalization ability. To solve this problem, EnvDrop (Tan et al., 2019) uses a speaker model to generate instructions for sampled trajecto- ries in unseen environments, but the generalization ability is not strong with limited vision-language understanding ability. Recently, VLN-BERT (Ma- jumdar et al., 2020) introduces a visio-linguistic model pretrained on Conceptual Captions (Sharma et al., 2018) dataset to learn from image-caption pairs, which are quite different from trajectory- …

Content truncated to 500 words