VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3377 - 3393 July 10-15, 2022 ©2022 Association for Computational Linguistics VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems Hung Le†‡§, Nancy F. Chen§, Steven C.H. Hoi†‡ †Salesforce Research Asia ‡Singapore Management University §Agency for Science, Technology and Research (A*STAR) {hungle,shoi}@salesforce.com, nfychen@i2r.a-star.edu.sg Abstract Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal vari- ance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Video- grounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGNMN first decomposes all language components in dialogues to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VGNMN can achieve promising performance on a challenging video-grounded dialogue benchmark as well as a video QA benchmark. 1 Introduction Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such as images, videos, and text. Extended from image-grounded tasks, e.g. (Antol et al., 2015), recently Jang et al. (2017); Lei et al. (2018) propose to use video as the ground- ing features. This modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames. Recently Alamri et al. (2019) further develop video- grounded language research into the dialogue do- main. In the proposed task, video-grounded dia- logues, the dialogue agent is required to answer questions about a video over multiple dialogue turns. Using Figure 1 as an example, to answer Video Caption: a boy and a man walk to the room. The boy carries his backpack while the man… Visual: ... Audio: ... Question: what is he doing while carrying it? find(he) find(it) Summarize() Where (a backpack) Where (a boy) when( while carryi ng it) Describe (what) Dialog History Question: how many people are in the video? Answer: there are a boy and a man Question: what is the boy doing? Answer: the boy walks downstairs and carries a backpack Predicted Answer: he is cleaning a mirror . ... ... Dialogue Understanding Video Understanding Figure 1: A sample video-grounded dialogue with a demonstration of a reasoning process questions correctly, a dialogue agent has to resolve references in dialogue context, e.g. “he” and “it”, and identify the original entity, e.g. “a boy" and “a backpack". Besides, the agent also needs to iden- tify the actions of these entities, e.g. “carrying a backpack” to retrieve information from the video. Current state-of-the-art approaches to video- grounded dialogue tasks, e.g. (Le et al., 2019b; Fan …

Content truncated to 500 words