Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1862–1872 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1862 Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation Vihan Jain∗ Gabriel Magalhaes∗ Alexander Ku∗ Ashish Vaswani Eugene Ie Jason Baldridge Google Research {vihan, gamaga, alexku, avaswani, eugeneie, jridge}@google.com Abstract Advances in learning and representations have reinvigorated work that connects language to other modalities. A particularly exciting direction is Vision-and-Language Navigation (VLN), in which agents interpret natural lan- guage instructions and visual scenes to move through environments and reach goals. De- spite recent progress, current research leaves unclear how much of a role language un- derstanding plays in this task, especially be- cause dominant evaluation metrics have fo- cused on goal completion rather than the se- quence of actions corresponding to the instruc- tions. Here, we highlight shortcomings of cur- rent metrics for the Room-to-Room dataset (Anderson et al., 2018b) and propose a new metric, Coverage weighted by Length Score (CLS). We also show that the existing paths in the dataset are not ideal for evaluating instruc- tion following because they are direct-to-goal shortest paths. We join existing short paths to form more challenging extended paths to cre- ate a new data set, Room-for-Room (R4R). Us- ing R4R and CLS, we show that agents that receive rewards for instruction fidelity outper- form agents that focus on goal completion. 1 Introduction In Vision-and-Language Navigation (VLN) tasks, agents must follow natural language navigation instructions through either simulated (Macmahon et al., 2006; Yan et al., 2018; Bisk et al., 2018; Shah et al., 2018), simulations of realistic (Blukis et al., 2018; Misra et al., 2018) and real environments (Anderson et al., 2018b; de Vries et al., 2018; Chen et al., 2019; Cirik et al., 2018), or actual physi- cal environments (Skoˇcaj et al., 2016; Thomason et al., 2018; Williams et al., 2018). Compared to other tasks involving co-grounding in visual and ∗Authors contributed equally. Figure 1: It’s the journey, not just the goal. To give lan- guage its due place in VLN, we compose paths in the R2R dataset to create longer, twistier R4R paths (blue). Under standard metrics, agents that head straight to the goal (red) are not penalized for ignoring the lan- guage instructions: for instance, SPL yields a perfect 1.0 score for the red and only 0.17 for the orange path. In contrast, our proposed CLS metric measures fidelity to the reference path, strongly preferring the agent with the orange path (0.87) over the red one (0.23). language modalities – such as image and video cap- tioning (Donahue et al., 2017; Fang et al., 2015; Vinyals et al., 2015; Wang et al., 2018; Yu et al., 2016), visual question answering (VQA) (Antol et al., 2015; Yang et al., 2016), and visual dia- log (Das et al., 2017) – VLN additionally requires agents to plan their actions, move, and dynamically respond to changes in their visual field. Photo-realistic simulations for VLN are espe- …

Content truncated to 500 words