NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6432–6442 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6432 NDH-FULL: Learning and Evaluating Navigational Agents on Full-Length Dialogue Hyounghun Kim Jialu Li UNC Chapel HillMohit Bansal {hyounghk, jialuli, mbansal}@cs.unc.edu Abstract Communication between human and mobile agents is getting increasingly important as such agents are widely deployed in our daily lives. Vision-and-Dialogue Navigation is one of the tasks that evaluate the agent’s ability to interact with humans for assistance and navi- gate based on natural language responses. In this paper, we explore the Navigation from Di- alogue History (NDH) task, which is based on the Cooperative Vision-and-Dialogue Nav- igation (CVDN) dataset, and present a state- of-the-art model which is built upon Vision- Language transformers. However, despite achieving competitive performance, we find that the agent in the NDH task is not evaluated appropriately by the primary metric – Goal Progress. By analyzing the performance mis- match between Goal Progress and other met- rics (e.g., normalized Dynamic Time Warping) from our state-of-the-art model, we show that NDH’s sub-path based task setup (i.e., navigat- ing partial trajectory based on its correspon- dent subset of the full dialogue) does not pro- vide the agent with enough supervision sig- nal towards the goal region. Therefore, we propose a new task setup called NDH-FULL which takes the full dialogue and the whole navigation path as one instance. We present a strong baseline model and show initial results on this new task. We further describe several approaches that we try, in order to improve the model performance (based on curriculum learning, pre-training, and data-augmentation), suggesting potential useful training methods on this new NDH-FULL task.1 1 Introduction With the increased number of intelligent agents being deployed in our daily lives, effective commu- nication between humans and agents is becoming more important. Natural language is one of the 1Our code and dataset are publicly available at: https: //github.com/hyounghk/NDH-FULL Should I stay on this floor or go down the stairs? Yeah, go down the stairs. And then I think you turn right but I can't really tell. Should I go left toward the door, or right around the corner? Go to the left, through that door. Target: picture Figure 1: One example in the CVDN dataset. Given target information, dialogues in blue text and red text sequentially, the human navigates the green path, blue path, and red path accordingly. most effective ways of communication due to its flexibility. Therefore, many efforts have been de- voted to exploring the potential of its application in several tasks. Vision-and-Language Navigation (VLN) is one of the tasks in which agents have to navigate to a goal location in the indoor or outdoor environment by following natural language instruc- tions (MacMahon et al., 2006; Tellex et al., 2011; Mei et al., 2016; Hermann et al., 2017; Brahmbhatt and Hays, 2017; Mirowski et al., 2018; Anderson et al., 2018; Misra et al., 2018; Blukis et al., 2019; Thomason et al., 2019; Nguyen and …

Content truncated to 500 words