Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1207–1221 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1207 Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation Wanrong Zhu†, Xin Eric Wang‡, Tsu-Jui Fu†, An Yan§, Pradyumna Narayana*, Kazoo Sone*, Sugato Basu*, William Yang Wang† †UC Santa Barbara, ‡UC Santa Cruz, §UC San Diego, *Google {wanrongzhu,tsu-juifu,william}@cs.ucsb.edu, xwang366@ucsc.edu, ayan@eng.ucsd.edu, {pradyn,sone,sugato}@google.com Abstract One of the most challenging topics in Nat- ural Language Processing (NLP) is visually- grounded language understanding and reason- ing. Outdoor vision-and-language navigation (VLN) is such a task where an agent fol- lows natural language instructions and navi- gates a real-life urban environment. Due to the lack of human-annotated instructions that illustrate intricate urban scenes, outdoor VLN remains a challenging task to solve. This pa- per introduces a Multimodal Text Style Trans- fer (MTST) learning approach and leverages external multimodal resources to mitigate data scarcity in outdoor navigation tasks. We first enrich the navigation data by transferring the style of the instructions generated by Google Maps API, then pre-train the navigator with the augmented external outdoor navigation dataset. Experimental results show that our MTST learning approach is model-agnostic, and our MTST approach significantly outper- forms the baseline models on the outdoor VLN task, improving task completion rate by 8.7% relatively on the test set. 1 1 Introduction A key challenge for Artificial Intelligence research is to go beyond static observational data and con- sider more challenging settings that involve dy- namic actions and incremental decision-making processes (Fenton et al., 2020). Outdoor vision- and-language navigation (VLN) is such a task, where an agent navigates in an urban environment by grounding natural language instructions in vi- sual scenes, as illustrated in Fig. 1. To generate a series of correct actions, the navigation agent must comprehend the instructions and reason through the visual environment. 1Our code and dataset is released at https://github. com/VegB/VLN-Transformer. Google Maps API Vanderbilt Ave turns right and becomes E 43rd St. Speaker You ' ll have a red brick building with a red awning on your right . Go forward until you reach the next intersection , and turn right. MTST model Turn right again and stop just past the orange and white construction barriers. Orient yourself so that the red deli awning is on your right. Turn left at the intersection. Figure 1: An outdoor VLN example with instructions generated by Google Maps API (ground truth), the Speaker model, and our MTST model. Tokens marked in red indicate incorrectly generated instructions, while the blue tokens suggest alignments with the ground truth. The orange bounding boxes show that the objects in the surrounding environment have been successfully injected into the style-modified instruction. Different from indoor navigation (Anderson et al., 2018; Wang et al., 2018; Fried et al., 2018; Wang et al., 2019; Ma et al., 2019a; Tan et al., 2019; Ma et al., 2019b; Ke et al., 2019), the outdoor navi- gation task takes place in urban environments that contain …

Content truncated to 500 words