Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3894 - 3904 July 10-15, 2022 ©2022 Association for Computational Linguistics Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering Jianguo Mao1,2∗, Wenbin Jiang3, Xiangdong Wang1, Zhifan Feng3, Yajuan Lyu3, Hong Liu1, Yong Zhu3 1Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China 2 University of Chinese Academy of Sciences, Beijing, China 3 Baidu Inc., Beijing, China {maojianguo20s,xdwang,hliu}@ict.ac.cn {jiangwenbin,fengzhifan,lvyajuan,zhuyong}@baidu.com Abstract Existing video question answering (video QA) models lack the capacity for deep video under- standing and flexible multistep reasoning. We propose for video QA a novel model which per- forms dynamic multistep reasoning between questions and videos. It creates video semantic representation based on the video scene graph composed of semantic elements of the video and semantic relations among these elements. Then, it performs multistep reasoning for better answer decision between the representations of the question and the video, and dynamically integrate the reasoning results. Experiments show the significant advantage of the proposed model against previous methods in accuracy and interpretability. Against the existing state- of-the-art model, the proposed model dramati- cally improves more than 4%/3.1%/2% on the three widely used video QA datasets, MSRVTT- QA, MSRVTT multi-choice, and TGIF-QA, and displays better interpretability by backtrac- ing along with the attention mechanisms to the video scene graphs. 1 Introduction Video question answering (video QA) aims to an- swer questions according to the given videos. It is usually defined as a classification task, where the most appropriate answer is chosen from a candidate list for the given question and video. Existing meth- ods for video QA conduct direct answering selec- tion based on the multimodal encoding of questions and videos (Jang et al., 2017; Lei et al., 2018, 2020). In recent years, researchers have proposed many optimization strategies for better performance in video question answering, e.g., designing delicate encoding mechanisms (Kim et al., 2020a; Nuamah, 2021; Gao et al., 2018; Li et al., 2019; Fan et al., 2019; Le et al., 2020; Jiang et al., 2020; Kim et al., 2020b; Seo et al., 2021), introducing video scene ∗This is joint work of CASICT and Baidu. Predicted Answer: ClipBERT: something Ours: ball Question (Q): what does a teenager end up on after his basketball throws miss the hoop as his friends watch by a gray house ? Question (Q): what are people throwing around ? Question (Q): what does the video show old footage of a man and also of ? Predicted Answer: ClipBERT: movie Ours: beach Predicted Answer: ClipBERT: trampoline Ours: ground Complexity：Easy Reasoning Process： -> Q: Identify answer type: Object (what) -> Q: Identify action: throwing around -> Q: Identify person: people -> I1-I4: Identify scene: a group of people -> I1-I4: Identify scene: people playing ball -> Choose the answer: ball Complexity：Medium Reasoning Process： -> Q: Identify answer type: Object (what) -> Q: Identify person: a …

Content truncated to 500 words