Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4392–4412, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4392 Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding Alexander Ku1∗ Peter Anderson1∗ Roma Patel2 Eugene Ie1 Jason Baldridge1 1Google Research 2Brown University {alexku, pjand, eugeneie, jridge}@google.com romapatel@brown.edu Abstract We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instruc- tions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more ref- erences to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and valida- tors. We establish baseline scores for mono- lingual and multilingual settings and multitask learning when including Room-to-Room anno- tations (Anderson et al., 2018b). We also pro- vide results for a model that learns from syn- chronized pose traces by focusing only on por- tions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for re- search on embodied language agents in simu- lated, photo-realistic environments. 1 Introduction Vision-and-Language Navigation (VLN) tasks re- quire computational agents to mediate the relation- ship between language, visual scenes and move- ment. Datasets have been collected for both indoor (Anderson et al., 2018b; Thomason et al., 2019b; Qi et al., 2020) and outdoor (Chen et al., 2019; Mehta et al., 2020) environments; success in these is based on clearly-defined, objective task completion rather than language or vision specific annotations. These VLN tasks fall in the Goldilocks zone: they can be tackled – but not solved – with current methods, and progress on them makes headway on real world grounded language understanding. We introduce Room-across-Room (RxR), a VLN dataset that addresses gaps in existing ones by (1) ∗First two authors contributed equally. Figure 1: RxR’s instructions are densely grounded to the visual scene by aligning the annotator’s virtual pose to their spoken instructions for navigating a path. including more paths that (2) counter known bi- ases in existing datasets, and (3) collecting an or- der of magnitude more instructions for (4) three languages (English, Hindi and Telugu) while (5) capturing annotators’ 3D pose sequences. As such, RxR includes dense spatiotemporal grounding for every instruction, as illustrated in Figure 1. We provide monolingual and multilingual base- line experiments using a variant of the Reinforced Cross-Modal Matching agent (Wang et al., 2019). Performance generally improves by using monolin- gual learning, and by using RxR’s follower paths as well as its guide paths. We also concatenate R2R and RxR annotations as a simple multitask strat- egy (Wang et al., 2020): the agent trained on both datasets obtains across the board improvements. RxR contains 126K instructions covering 16.5K sampled guide paths and 126K human follower 4393 demonstration paths. The dataset is available.1 We plan to release a test evaluation server, our annota- tion tool, and code for all experiments. 2 Motivation A …

Content truncated to 500 words