On the Evaluation of Vision-and-Language Navigation Instructions

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1302–1316 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1302 On the Evaluation of Vision-and-Language Navigation Instructions Ming Zhao Peter Anderson Vihan Jain Su Wang Alexander Ku Jason Baldridge Eugene Ie Google Research {astroming, pjand, vihan, wangsu, alexku, jridge, eugeneie} @google.com Abstract Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automat- ically generated navigation instructions. How- ever, existing instruction generators have not been comprehensively evaluated, and the au- tomatic evaluation metrics used to develop them have not been validated. Using human wayfinders, we show that these generators per- form on par with or only slightly better than a template-based generator and far worse than human instructors. Furthermore, we discover that BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded naviga- tion instructions. To improve instruction eval- uation, we propose an instruction-trajectory compatibility model that operates without ref- erence instructions. Our model shows the high- est correlation with human wayfinding out- comes when scoring individual instructions. For ranking instruction generation systems, if reference instructions are available we recom- mend using SPICE. 1 Introduction Generating route instructions is a long studied prob- lem with clear practical applications (Richter and Klippel, 2005). Whereas earlier work sought to create instructions for human wayfinders, recent work has focused on using instruction-generation models to improve the performance of agents that follow instructions given by people. In the con- text of Vision-and-Language Navigation (VLN) datasets such as Room-to-Room (R2R) (Anderson et al., 2018b), models for generating navigation instructions have improved agents’ wayfinding per- formance in at least two ways: (1) by synthesizing new instructions for data augmentation (Fried et al., 2018; Tan et al., 2019), and (2) by fulfilling the role of a probabilistic speaker in a pragmatic reasoning setting (Fried et al., 2018). Such data augmentation is so effective that it is nearly ubiquitous in the best Visual Encoder Instruction Encoder Compatibility Matrix S(i, j) Dot Product i j Walk up stairs and enter the first room on the left. Walk towards the end of the bedroom and stop inside the bathroom. hv hw Instruction Representation Trajectory Representation Figure 1: Proposed dual encoder instruction-trajectory compatibility model. Navigation instructions and tra- jectories (sequences of panoramic images and view an- gles) are projected into a shared latent space. The in- dependence between the encoders facilitates learning using both contrastive and classification losses. performing agents (Wang et al., 2019; Huang et al., 2019; Li et al., 2019). To make further advances in the generation of visually-grounded navigation instructions, accurate evaluation of the generated text is essential. How- ever, the performance of existing instruction gen- erators has not yet been evaluated using human wayfinders, and the efficacy of the automated eval- uation metrics used to develop them has not been established. This paper addresses both gaps. To establish benchmarks for navigation instruc- tion generation, we evaluate existing English mod- els (Fried et al., 2018; Tan et al., 2019) using human …

Content truncated to 500 words