Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6551–6557 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6551 Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation Ronghang Hu1, Daniel Fried1, Anna Rohrbach1, Dan Klein1, Trevor Darrell1, Kate Saenko2 1University of California, Berkeley 2Boston University {ronghang,dfried,anna.rohrbach,klein}@cs.berkeley.edu, trevor@eecs.berkeley.edu,saenko@bu.edu Abstract Vision-and-Language Navigation (VLN) re- quires grounding instructions, such as turn right and stop at the door, to routes in a vi- sual environment. The actual grounding can connect language to the environment through multiple modalities, e.g. stop at the door might ground into visual objects, while turn right might rely only on the geometric structure of a route. We investigate where the natural lan- guage empirically grounds under two recent state-of-the-art VLN models. Surprisingly, we discover that visual features may actually hurt these models: models which only use route structure, ablating visual features, outperform their visual counterparts in unseen new envi- ronments on the benchmark Room-to-Room dataset. To better use all the available modal- ities, we propose to decompose the grounding procedure into a set of expert models with ac- cess to different modalities (including object detections) and ensemble them at prediction time, improving the performance of state-of- the-art models on the VLN task. 1 Introduction The Vision-and-Language Navigation (VLN) task (Anderson et al., 2018) requires an agent to nav- igate to a particular location in a real-world en- vironment, following complex, context-dependent instructions written by humans (e.g. go down the second hallway on the left, enter the bedroom and stop by the mirror). The agent must navigate through the environment, conditioning on the in- struction as well as the visual imagery that it ob- serves along the route, to stop at the location spec- ified by the instruction (e.g. the mirror). Recent state-of-the-art models (Wang et al., 2018; Fried et al., 2018b; Ma et al., 2019) have demonstrated large gains in accuracy on the VLN task. However, it is unclear which modality these go past the couch … Instruction: Route Structure: Non- Visual Agent: 39.7% success go past the couch … Instruction: Route Structure and Visual Appearance: Visual Agent: 40.5% success Prior Work go past the couch … Instruction: Route Structure and Object Detections: Object- based Agent: 41.6% success couch door lamp chair stairs Mixture-of- Experts Agent: 51.9% success Figure 1: We factor the grounding of language instruc- tions into visual appearance, route structure, and object detections using a mixture-of-experts approach. substantial increases in task metrics can be at- tributed to, and, in particular, whether the gains in performance are due to stronger grounding into visual context or e.g. simply into the discrete, geo- metric structure of possible routes, such as turning left or moving forward (see Fig. 1, top vs. middle). First, we analyze to what extent VLN mod- els ground language into visual appearance and route structure by training versions of two state-of- the-art models without visual features, using the benchmark Room-to-Room (R2R) dataset (Ander- son et al., …

Content truncated to 500 words