Dual Attention Networks for Visual Reference Resolution in Visual Dialog

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2024–2033, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2024 Dual Attention Networks for Visual Reference Resolution in Visual Dialog Gi-Cheon Kang Seoul National University chonkang@snu.ac.kr Jaeseo Lim Seoul National University jaeseolim@snu.ac.kr Byoung-Tak Zhang Seoul National University Surromind Robotics btzhang@snu.ac.kr Abstract Visual dialog (VisDial) is a task which re- quires a dialog agent to answer a series of questions grounded in an image. Unlike in vi- sual question answering (VQA), the series of questions should be able to capture a tempo- ral context from a dialog history and utilizes visually-grounded information. Visual refer- ence resolution is a problem that addresses these challenges, requiring the agent to resolve ambiguous references in a given question and to find the references in a given image. In this paper, we propose Dual Attention Net- works (DAN) for visual reference resolution in VisDial. DAN consists of two kinds of at- tention modules, REFER and FIND. Specif- ically, REFER module learns latent relation- ships between a given question and a dia- log history by employing a multi-head atten- tion mechanism. FIND module takes image features and reference-aware representations (i.e., the output of REFER module) as input, and performs visual grounding via bottom-up attention mechanism. We qualitatively and quantitatively evaluate our model on VisDial v1.0 and v0.9 datasets, showing that DAN out- performs the previous state-of-the-art model by a significant margin. 1 Introduction Thanks to the recent progresses in natural lan- guage processing and computer vision, there has been an extensive amount of effort towards devel- oping a cognitive agent that jointly understand nat- ural language and vision information. Over the last few years, vision-language tasks such as im- age captioning (Xu et al., 2015) and visual ques- tion answering (VQA) (Antol et al., 2015; Ander- son et al., 2018) have provided a testbed for devel- oping a cognitive agent. However, the agent per- forming these tasks still has a long way to go to be used in real-world applications (e.g., aiding vi- sually impaired users, interacting with humanoid robots) in that it does not consider the continuous interaction over time. Specifically, the interaction in image captioning is that the agent simply talks to human about visual content, without any input from human. While the VQA agent takes a ques- tion as input, it is required to answer a single ques- tion about a given image. Visual dialog (VisDial) (Das et al., 2017) task has been introduced as a generalized version of VQA. A dialog agent needs to answer a series of questions such as “How many people are in the image?”, “Are they indoors or outside?”, utilizing not only visually-grounded information but also contextual information from a dialog history. To address these two challenges, researchers have re- cently tackled a problem called visual reference resolution in VisDial. The problem of visual ref- erence resolution is to resolve ambiguous expres- sions …

Content truncated to 500 words