VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator

Findings of the Association for Computational Linguistics: ACL 2022, pages 1984 - 1994 May 22-27, 2022 c⃝2022 Association for Computational Linguistics VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator Ayush Shrivastava1∗, Karthik Gopalakrishnan2, Yang Liu2, Robinson Piramuthu2, Gokhan Tür2, Devi Parikh1, Dilek Hakkani-Tür2 1Georgia Tech, 2Amazon Alexa AI {ayshrv, parikh}@gatech.edu {karthgop, yangliud, robinpir, gokhatur, hakkanit}@amazon.com Abstract Interactive robots navigating photo-realistic en- vironments need to be trained to effectively leverage and handle the dynamic nature of dia- logue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VIS- ITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) iden- tify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical in- sights and improve performance on CVDN. VISITRON’s ability to identify when to interact leads to a natural generalization of the game- play mode introduced by Roman et al. (2020) for enabling the use of such models in differ- ent environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric. 1 Introduction Large pre-trained Transformer-based language models (Vaswani et al., 2017) are ubiquitous in natural language processing (NLP) and have per- formed very well in interactive settings such as open-domain (Gopalakrishnan et al., 2019; Huang et al., 2020) and task-oriented dialogue (Kim et al., 2020). The success of Transformers and the pre- train/fine-tune paradigm in NLP has also inspired their adoption in vision-and-language research, with cross-modal representations being learned (Li et al., 2020) and utilized towards tasks like image and object captioning, visual question answering, visual commonsense reasoning and visual dialogue. ∗ Work done as an intern at Amazon Alexa AI. Code available at: www.github.com/alexa/visitron Ask question Dialogue History NAVIGATOR Enter the doorway on the right, the one between the two long tables with cloths on them. Goal Room contains Picture Do I keep going past the table or is the picture the one above the table to my right? Navigate Role of GUIDE GUIDE Question Next 5 steps Answer Role of NAVIGATOR Do I keep going past the table or is the picture the one above the table to my right? Do I keep going past the table or is the picture the one above the table to my right? Enter the doorway on the right, the one between the two long tables with cloths on them. … Figure 1: Cooperative Vision-and-Dialog Navigation (CVDN) with Dynamic Question-Asking Vision-and-language navigation (VLN) is a chal- lenging cross-modal research task in which agents need to learn to navigate in response to natural language instructions in simulated photo-realistic environments. VLN has been studied extensively with the advent of the Room-to-Room (R2R) dataset (Anderson et al., 2018b) and there has been growing interest recently in pushing the …

Content truncated to 500 words