Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7606 - 7623 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions Jing Gu1 Eliana Stefani1 Qi Wu2 Jesse Thomason3 Xin Eric Wang1 1University of California, Santa Cruz 2The University of Adelaide 3University of Southern California {jgu110,estefani,xwang366}@ucsc.edu qi.wu01@adelaide.edu.au, jessetho@usc.edu Abstract A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the envi- ronment, and perform real-world tasks. Vision- and-Language Navigation (VLN) is a funda- mental and interdisciplinary research topic to- wards this goal, and receives increasing atten- tion from natural language processing, com- puter vision, robotics, and machine learning communities. In this paper, we review contem- porary studies in the emerging field of VLN, covering tasks, evaluation metrics, methods, etc. Through structured analysis of current progress and challenges, we highlight the lim- itations of current VLN and opportunities for future work. This paper serves as a thorough reference for the VLN research community.1 1 Introduction Humans communicate with each other using nat- ural language to issue tasks and request help. An agent that can understand human language and nav- igate intelligently would significantly benefit hu- man society, both personally and professionally. Such an agent can be spoken to in natural lan- guage, and would autonomously execute tasks such as household chores indoors, repetitive delivery work outdoors, or work in hazardous conditions following human commands (bridge inspection; fire-fighting). Scientifically, developing such an agent explores how an artificial agent interprets natural language from humans, perceives its visual environment, and utilizes that information to navi- gate to complete a task successfully. Vision-and-Language Navigation (VLN) (An- derson et al., 2018b; Chen et al., 2019; Thoma- son et al., 2019b) is an emerging research field that aims to build such an embodied agent that can in1We also release a Github repo to keep track of advances VLN: https://github.com/eric-ai-lab/ awesome-vision-language-navigation Natural Language Communication Observation Action Observation Action Agent Oracle Environment Figure 1: The agent and oracle discuss the VLN task in natural language. Both observe and interact with the navigable environment to accomplish a task. communicate with humans in natural language and navigate in real 3D environments. VLN extends vi- sual navigation in both simulated (Zhu et al., 2017; Mirowski, 2019) and real environments (Mirowski et al., 2018) with natural language communication. As illustrated in Figure 1, VLN is a task that in- volves the oracle (frequently a human), the agent, and the environment. The agent and the oracle communicate in natural language. The agent may ask for guidance and the oracle could respond. The agent navigates and interacts with the environment to complete the task according to the instructions re- ceived and the environment observed. Meanwhile, the oracle observes the environment and agent sta- tus, and may interact with the environment to help the agent. Since the development and release of works such as Room-to-Room (R2R) (Anderson et al., …

Content truncated to 500 words