Video Dialog via Progressive Inference and Cross-Transformer

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2109–2118, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2109 Video Dialog via Progressive Inference and Cross-Transformer Weike Jin1, Zhou Zhao1∗, Mao Gu1, Jun Xiao1, Furu Wei2, Yueting Zhuang1 1Zhejiang University, Hangzhou 2Microsoft Research Asia, Beijing {weikejin,zhaozhou,21821134,junx,yzhuang}@zju.edu.cn fuwei@microsoft.com Abstract Video dialog is a new and challenging task, which requires the agent to answer questions combining video information with dialog his- tory. And different from single-turn video question answering, the additional dialog his- tory is important for video dialog, which often includes contextual information for the ques- tion. Existing visual dialog methods mainly use RNN to encode the dialog history as a single vector representation, which might be rough and straightforward. Some more ad- vanced methods utilize hierarchical structure, attention and memory mechanisms, which still lack an explicit reasoning process. In this paper, we introduce a novel progressive in- ference mechanism for video dialog, which progressively updates query information based on dialog history and video content until the agent think the information is sufficient and unambiguous. In order to tackle the multi- modal fusion problem, we propose a cross- transformer module, which could learn more fine-grained and comprehensive interactions both inside and between the modalities. And besides answer generation, we also consider question generation, which is more challeng- ing but significant for a complete video dialog system. We evaluate our method on two large- scale datasets, and the extensive experiments show the effectiveness of our method. 1 Introduction Visual dialog can be seen as an extension of the visual question answering (VQA) (Antol et al., 2015; Yu et al., 2017). Unlike visual question answering, in which each question is asked inde- pendently, visual dialog requires the agent to an- swer multi-round questions and the previous round of question-answer pairs form the dialog history. Currently, most of the existing visual dialog ap- proaches mainly focus on image dialog (Das et al., ∗Zhou Zhao is the corresponding author. Q1: What does the person A1: Pepper. Q2: What color is the plate A2: Yellow. Q3: What does the person A3: He makes a sauce. Q4: What color is it? A4: Green. Q: What does the person d A: He mixes the sauce wit What does the person add to the water? Pepper. What color is the plate with meat? Yellow. What does the person make in the plastic container? He makes a sauce. What color is it? Green. What does the person do with it? He mixes the sauce with meat. Dialog History Progressive Inference What does the person do with it? Cross-Transformer Generator Answer Question Refined Query with Context Video Information Stop sauce in plastic container color green meat in plate color yellow Figure 1: An illustration of our video dialog model with progressive inference and cross-transformer. 2017a; Seo et al., 2017; Kottur et al., 2018; Lee et al., 2018). Video dialog is still less explored. For …

Content truncated to 500 words