Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6463–6474 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6463 Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog Zhe Gan1, Yu Cheng1, Ahmed El Kholy1, Linjie Li1, Jingjing Liu1, Jianfeng Gao2 1Microsoft Dynamics 365 AI Research, 2Microsoft Research {zhe.gan, yu.cheng, ahmed.eikholy, lindsey.li, jingjl, jfgao}@microsoft.com Abstract This paper presents a new model for vi- sual dialog, Recurrent Dual Attention Net- work (ReDAN), using multi-step reasoning to answer a series of questions about an im- age. In each question-answering turn of a di- alog, ReDAN infers the answer progressively through multiple reasoning steps. In each step of the reasoning process, the semantic rep- resentation of the question is updated based on the image and the previous dialog history, and the recurrently-refined representation is used for further reasoning in the subsequent step. On the VisDial v1.0 dataset, the pro- posed ReDAN model achieves a new state-of- the-art of 64.47% NDCG score. Visualization on the reasoning process further demonstrates that ReDAN can locate context-relevant vi- sual and textual clues via iterative refinement, which can lead to the correct answer step-by- step. 1 Introduction There has been a recent surge of interest in de- veloping neural network models capable of under- standing both visual information and natural lan- guage, with applications ranging from image cap- tioning (Fang et al., 2015; Vinyals et al., 2015; Xu et al., 2015) to visual question answering (VQA) (Antol et al., 2015; Fukui et al., 2016; Anderson et al., 2018). Unlike VQA, where the model can answer a single question about an im- age, a visual dialog system (Das et al., 2017a; De Vries et al., 2017; Das et al., 2017b) is designed to answer a series of questions regarding an image, which requires a comprehensive understanding of both the image and previous dialog history. Most previous work on visual dialog rely on at- tention mechanisms (Bahdanau et al., 2015; Xu et al., 2015) to identify specific regions of the im- age and dialog-history snippets that are relevant to the question. These attention models measure the relevance between the query and the attended image, as well as the dialog context. To generate an answer, either a discriminative decoder is used for ranking answer candidates, or a generative de- coder is trained for synthesizing an answer (Das et al., 2017a; Lu et al., 2017). Though promis- ing results have been reported, these models of- ten fail to provide accurate answers, especially in cases where answers are confined to particular im- age regions or dialog-history snippets. One hypothesis for the cause of failure is the inherent limitation of single-step reasoning ap- proach. Intuitively, after taking a first glimpse of the image and the dialog history, readers often re- visit specific sub-areas of both image and text to obtain a better understanding of the multimodal context. Inspired by this, we propose a Recur- rent Dual Attention Network (ReDAN) that ex- ploits multi-step reasoning for …

Content truncated to 500 words