BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1846–1859, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1846 BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues Hung Le†§∗, Doyen Sahoo‡, Nancy F. Chen§, Steven C.H. Hoi†‡ † Singapore Management University hungle.2018@smu.edu.sg ‡ Salesforce Research Asia {dsahoo,shoi}@salesforce.com §Institute for Infocomm Research, A*STAR nfychen@i2r.a-star.edu.sg Abstract Video-grounded dialogues are very challeng- ing due to (i) the complexity of videos which contain both spatial and temporal variations, and (ii) the complexity of user utterances which query different segments and/or dif- ferent objects in videos over multiple dia- logue turns. However, existing approaches to video-grounded dialogues often focus on superficial temporal-level visual cues, but ne- glect more fine-grained spatial signals from videos. To address this drawback, we pro- pose Bi-directional Spatio-Temporal Learning (BiST), a vision-language neural framework for high-resolution queries in videos based on textual cues. Specifically, our approach not only exploits both spatial and temporal-level information, but also learns dynamic informa- tion diffusion between the two feature spaces through spatial-to-temporal and temporal-to- spatial reasoning. The bidirectional strategy aims to tackle the evolving semantics of user queries in the dialogue setting. The retrieved visual cues are used as contextual information to construct relevant responses to the users. Our empirical results and comprehensive qual- itative analysis show that BiST achieves com- petitive performance and generates reasonable responses on a large-scale AVSD benchmark. We also adapt our BiST models to the Video QA setting, and substantially outperform prior approaches on the TGIF-QA benchmark. 1 Introduction A video-grounded dialogue agent aims to converse with humans not only based on signals from natu- ral language but also from other modalities such as sound and vision of the input video. Recent efforts (Alamri et al., 2018; Sanabria et al., 2019; Alamri et al., 2019) consider video-grounded dialogues as an extension of video Question-Answering (QA) ∗This work was mostly done when Hung Le was an intern at Salesforce Research Asia, Singapore. H: How many people are in the video? H: What room is he in? H: Does he enter the kitchen? H: What does he do before he sits down? H: Is he happy or sad? A: There is just one person A: The man is in the kitchen A: No he is already in the kitchen A: He closes the cabinet and then walks to the table and pulls the chair out to sit down A: He appears to be neutral in expression H: hello , what is the first thing he does ? H: where does he put it ? H: what does he do after that ? A: he takes off his long sleeve shirt A: he sets it on the table in front of him . A: he opens the door and sits down on the ground . Figure 1: Examples of video-grounded dialogues from the benchmark datasets of Audio-Visual Scene Aware Dialogues (AVSD) challenge (Alamri et al., 2018, 2019). H: human, A: the dialogue agent. (Tapaswi et al., 2016; Jang et al., …

Content truncated to 500 words