Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612–5623 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5612 Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems Hung Le1,2, Doyen Sahoo1, Nancy F. Chen2, Steven C.H. Hoi1,3 1Singapore Management University 2Institute of Inforcomm Research (I2R), Singapore 3Salesforce Research Asia {hungle.2018,doyens}@smu.edu.sg nfychen@i2r.a-star.edu.sg,chhoi@smu.edu.sg Abstract Developing Video-Grounded Dialogue Sys- tems (VGDS), where a dialogue is conducted based on visual and audio aspects of a given video, is significantly more challenging than traditional image or text-grounded dialogue systems because (1) feature space of videos span across multiple picture frames, making it difficult to obtain semantic information; and (2) a dialogue agent must perceive and process information from different modalities (audio, video, caption, etc.) to obtain a comprehensive understanding. Most existing work is based on RNNs and sequence-to-sequence architec- tures, which are not very effective for captur- ing complex long-term dependencies (like in videos). To overcome this, we propose Mul- timodal Transformer Networks (MTN) to en- code videos and incorporate information from different modalities. We also propose query- aware attention through an auto-encoder to extract query-aware features from non-text modalities. We develop a training procedure to simulate token-level decoding to improve the quality of generated responses during in- ference. We get state of the art performance on Dialogue System Technology Challenge 7 (DSTC7). Our model also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. 1 Introduction A video-grounded dialogue system (VGDS) gen- erates appropriate conversational response to queries of humans, by not only keeping track of the relevant dialogue context, but also under- standing the relevance of the query in the con- text of a given video (knowledge grounded in a video) (Hori et al., 2018). An example dia- logue exchange can be seen in Figure 1. Devel- oping such systems has recently received interest from the research community (e.g. DSTC7 chal- lenge (Yoshino et al., 2018)). This task is much C: a man is standing in a kitchen putting groceries away. He closes the cabinet when finished, walks over to a table and pulls out a chair and sits down. S: a man puts away his groceries and then sits at a kitchen table and stares out the window. Q1: how many people are in the video? A1: there is just one person Q2: is there sound to the video? ...A2: yes there is audio but no one is talking Q10: is he happy or sad? A10: he appears to be neutral in expression Figure 1: A sample dialogue from the DSTC7 Video Scene-aware Dialogue training set with 4 example video scenes. C: Video Caption, S: Video Summary, Qi: ith-turn question, Ai: ith-turn answer more challenging than traditional text-grounded or image-grounded dialogue systems because: (1) feature space of videos is larger and more complex than text-based or image-based features because of diverse information, such as background noise, human speech, flow of actions, etc. across mul- tiple video frames; and (2) a conversational …

Content truncated to 500 words