FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4332 - 4340 July 10-15, 2022 ©2022 Association for Computational Linguistics FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation Zi-Yi Dou, Nanyun Peng Department of Computer Science University of California, Los Angeles {zdou,violetpeng}@cs.ucla.edu Abstract The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model. However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower. In this paper, we present FOAM, a FOllower-Aware speaker Model that is constantly updated given the fol- lower feedback, so that the generated instruc- tions can be more suitable to the current learn- ing state of the follower. Specifically, we opti- mize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data. Ex- perimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings. Analyses also reveal that our generated instructions are of higher quality than the baselines.1 1 Introduction The task of vision-and-language navigation (VLN) requires an agent to navigate in a real-world envi- ronment given natural language instructions. In VLN, one of the major challenges is the lack of training data. To alleviate the issue, speaker- follower models (Fried et al., 2018b) have been pro- posed. Specifically, in the speaker-follower models, an instruction-follower agent is trained to follow a provided natural language instruction to complete a specified goal, and a speaker model learns to model how humans describe routes and synthesize new instructions so as to create more training data for the follower. While speaker-augmented data is widely used in VLN (Fried et al., 2018b; Wang et al., 2019; Ma et al., 2019; Tan et al., 2019; Zhu et al., 2020a; 1Code is available at https://github.com/ PlusLabNLP/follower_aware_speaker. Sampled Routes Walk past the dining room table and chairs and … Generated Instructions Ground-truth routes Continue forward until you can climb the three steps to your right ... Ground-truth Routes Human Instructions (b) train (a) generate (c) evaluate (d) update Speaker Follower Figure 1: Many of the previous methods use the speaker to generate instructions from sampled routes and train the follower. FOAM (in red) further obtains feedback from the follower on labeled data and updates the speaker accordingly. Hao et al., 2020; Wang et al., 2021; Chen et al., 2021), most of the previous methods focus on im- proving the follower navigation model. In contrast, how to improve the speaker model to generate data of higher quality is underexplored. In this line of research, Fried et al. (2018a) build a pragmatic speaker that can synthesize instructions based on how the follower may interpret the instructions; Tan et al. (2019) propose to randomly add noise into the environments when generating instructions, so that the noisy environments can …

Content truncated to 500 words