In-the-Wild Video Question Answering

Proceedings of the 29th International Conference on Computational Linguistics, pages 5613–5635 October 12–17, 2022. 5613 WildQA: In-the-Wild Video Question Answering Santiago Castro∗ Naihao Deng∗ Pingxuan Huang∗ Mihai Burzo Rada Mihalcea University of Michigan – Ann Arbor, USA {sacastro, dnaihao, pxuanh, mburzo, mihalcea}@umich.edu Q: What is the man using as tools? A: A saw and handaxe E: 00:00 00:37 00:51 Figure 1: An example from our WildQA dataset, showing a question (Q), an answer (A), and evidence (E) that supports the answer. The corresponding part of the videos is provided as evidence for the question. Abstract Existing video understanding datasets mostly focus on human interactions, with little at- tention being paid to the "in the wild" set- tings, where the videos are recorded outdoors. We propose WILDQA, a video understanding dataset of videos recorded in outside settings. In addition to video question answering (Video QA), we also introduce the new task of identi- fying visual support for a given question and answer (Video Evidence Selection). Through evaluations using a wide range of baseline mod- els, we show that WILDQA poses new chal- lenges to the vision and language research com- munities. The dataset is available at https: //lit.eecs.umich.edu/wildqa/. 1 Introduction Video understanding plays an important role in the development of intelligent AI systems, as it enables the effective processing of different modalities of information (Li et al., 2021a). Various tasks have been proposed to examine the ability of models’ to understand videos, including video question an- swering (Video QA), video captioning, and fill-in- the-blank tasks (Xu et al., 2017; Tran et al., 2016; *: Equal contribution Castro et al., 2022). Recent years have witnessed significant progress in video understanding, includ- ing new benchmarks (Tapaswi et al., 2016; Grau- man et al., 2021) as well as advanced sophisticated benchmarksmodels (Jin et al., 2019; Radford et al., 2021). There are however several drawbacks associated with existing video understanding research. First, existing video understanding benchmarks focus on common human activities as typically appear- ing in cooking videos (Zhu et al., 2017) or in movies (Tapaswi et al., 2016), leading to a lim- ited set of video domains. Second, most video understanding benchmarks adopt a multiple-choice format, where models select an answer from a set of candidates (Jang et al., 2017; Castro et al., 2020). Models trained under such a setting cannot be used in real-life applications because candidate an- swers are not provided (Castro et al., 2022). Third, videos included in existing benchmarks are typi- cally short (Kim et al., 2016), and the performance of models on longer videos is not well studied. We address these challenges in our dataset con- struction process. First, we propose the WILDQA dataset in which we collect “in the wild” videos that are recorded in the outside world, going beyond daily human activities. Figure 2 shows the differ- 5614 Example from MovieQA (multiple-choice) Q: How does E.T. show his happiness that he is finally returning home? A: His heart lights up. Example from WildQAours (open-ended) Q: What sort …

Content truncated to 500 words