Video Question Answering: Datasets, Algorithms and Challenges

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6439 - 6455 December 7-11, 2022 ©2022 Association for Computational Linguistics Video Question Answering: Datasets, Algorithms and Challenges Yaoyao Zhong1,3∗, Junbin Xiao1,2∗, Wei Ji1,2∗, Yicong Li1, Weihong Deng3†, Tat-Seng Chua1,2 1National University of Singapore, Singapore 2Sea-NExT Joint Lab, Singapore 3Beijing University of Posts and Telecommunications, Beijing, China {zhongyaoyao,whdeng}@bupt.edu.cn, junbin@comp.nus.edu.sg, liyicong@u.nus.edu, {jiwei,dcscts}@nus.edu.sg Abstract This survey aims to organize the recent ad- vances in video question answering (VideoQA) and point towards future directions. We firstly categorize the datasets into: 1) nor- mal VideoQA, multi-modal VideoQA and knowledge-based VideoQA, according to the modalities invoked in the question-answer pairs, and 2) factoid VideoQA and inference VideoQA, according to the technical challenges in comprehending the questions and deriving the correct answers. We then summarize the VideoQA techniques, including those mainly designed for Factoid QA (such as the early spatio-temporal attention-based methods and the recent Transformer-based ones) and those targeted at explicit relation and logic infer- ence (such as neural modular networks, neu- ral symbolic methods, and graph-structured methods). Aside from the backbone tech- niques, we also delve into specific models and derive some common and useful insights either for video modeling, question answer- ing, or for cross-modal correspondence learn- ing. Finally, we present the research trends of studying beyond factoid VideoQA to inference VideoQA, as well as towards the robustness and interpretability. Additionally, we maintain a repository, https://github.com/VRU-NExT/ VideoQA, to keep trace of the latest VideoQA papers, datasets, and their open-source imple- mentations if available. With these efforts, we strongly hope this survey could shed light on the follow-up VideoQA research. 1 Introduction Recent years have witnessed a flourish of research in vision-language understanding (Xu et al., 2016; Chen et al., 2017; Antol et al., 2015; Chen et al., 2018; Jang et al., 2017), of which, video Ques- tion Answering (VideoQA) is one of the most prominent, given its promise to develop interac- tive AI to communicate with the dynamic visual ∗ The first three authors contribute equally to this work. † Corresponding authors. world via natural languages. Despite the popularity, VideoQA remains one of the greatest challenges, because it demands the models to comprehensively understand the videos to correctly answer questions. The questions involve not only the recognition of visual objects, actions, activities and events, but also the inference of their semantic, spatial, tempo- ral, and causal relationships (Xu et al., 2017; Jang et al., 2017; Shang et al., 2019, 2021; Yang et al., 2021b; Xiao et al., 2021, 2022a). To tackle the challenges, techniques such as spatio-temporal attention (Jang et al., 2017), motion-appearance memory (Gao et al., 2018), and spatio-temporal or hierarchical graph models (Cherian et al., 2022; Xiao et al., 2022a) have been proposed and demonstrated their effectiveness on different VideoQA datasets. However, we find that the datasets, the defined challenges, and the cor- responding algorithms are varied and a bit messy. There is a lack of a meaningful survey to categorize the datasets and to organize the technique …

Content truncated to 500 words