Visual-Textual Alignment for Graph Inference in Visual Dialog

Proceedings of the 28th International Conference on Computational Linguistics, pages 1874–1885 Barcelona, Spain (Online), December 8-13, 2020 1874 Visual-Textual Alignment for Graph Inference in Visual Dialog Tianling Jiang School of Computer Science and Technology Soochow University tljiang@stu.suda.edu.cn Chunping Liu School of Computer Science and Technology Soochow University cpliu@suda.edu.cn Ji Yi School of Computer Science and Technology Soochow University jiyi@suda.edu.cn Hailin Shao School of Computer Science and Technology Soochow University 20184227001@stu.suda.edu.cn Abstract As a conversational intelligence task, visual dialog entails answering a series of questions grounded in an image, using the dialog history as context. To generate correct answers, the comprehension of the semantic dependencies among implicit visual and textual contents is crit- ical. Prior works usually ignored the underlying relation and failed to infer it reasonably. In this paper, we propose a Visual-Textual Alignment for Graph Inference (VTAGI) network. Com- pared with other approaches, it makes up the lack of structural inference in visual dialog. The whole system consists of two modules, Visual and Textual Alignment (VTA) and Visual Graph Attended by Text (VGAT). Specially, the VTA module aims at representing an image with a set of integrated visual regions and corresponding textual concepts, reflecting certain semantics. The VGAT module views the visual features with semantic information as observed nodes and each node learns the relationship with others in visual graph. We also qualitatively and quanti- tatively evaluate the model on VisDial v1.0 dataset, showing our VTAGI outperforms previous state-of-the-art models. 1 Introduction Cross-modal semantic understanding has become an attractive challenge in natural language processing and computer vision, inspiring many tasks such as image captioning(Xu et al., 2015; Vinyals et al., 2015) and visual question answering (VQA)(Antol et al., 2015; Anderson et al., 2018; Shimizu et al., 2018). However, in these missions, the co-reference between vision and language is usually performed in a single round and they do not have many interactions with human over a period of time. In 2017, Das et al. introduced a continuous conversational task, visual dialog(Das et al., 2017). This task needs an AI agent to answer a sequence of questions based on visually-grounded information and contextual information from a dialog history. Recently, a manual investigation(Kim et al., 2020) on the Visual Dialog dataset (VisDial) tried to figure out how many questions can be answered with images and how many of them need conversation history to be answered. The investigation shows that around 80% of the questions can be answered with images and about 20% of the questions need the knowledge from dialog history. Therefore, one of the key chal- lenges in visual dialog is how to effectively utilize these underlying contents in the textual and visual information, i.e., input questions, dialog history and input image. In previous works, such as RvA(Niu et al., 2019) and DAN(Kang et al., 2019), both tended to explicitly reason over past dialog interactions by referring back to previous references, but they ignored the underlying relational structure which con- tributes to dialog inference. Nowadays, researchers have attempted to consider the …

Content truncated to 500 words