LifeQA: A Real-life Dataset for Video Question Answering

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4352–4358 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4352 LifeQA: A Real-life Dataset for Video Question Answering Santiago Castro, Mahmoud Azab, Jonathan C. Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, Rada Mihalcea University of Michigan {sacastro, mazab, stroud, cnoujaim, ruoyaow, jiadeng, mihalcea}@umich.edu Abstract We introduce LifeQA, a benchmark dataset for video question answering that focuses on day-to-day real-life situations. Current video question answering datasets consist of movies and TV shows. However, it is well-known that these visual domains are not representative of our day-to-day lives. Movies and TV shows, for example, benefit from professional camera movements, clean editing, crisp audio recordings, and scripted dialog between professional actors. While these domains provide a large amount of data for training models, their properties make them unsuitable for testing real-life question answering systems. Our dataset, by contrast, consists of video clips that represent only real-life scenarios. We collect 275 such video clips and over 2.3k multiple-choice questions. In this paper, we analyze the challenging but realistic aspects of LifeQA, and we apply several state-of-the-art video question answering models to provide benchmarks for future research. The full dataset is publicly available at https://lit.eecs.umich.edu/lifeqa/. Keywords: natural language processing, question answering, video question answering, computer vision What is the name of the younger girl? A. Caitlin B. Lucy C. Jane D. Cindy How many people are playing? A. 2 B. 4 C. 3 D. 1 Caitlin, are you gonna be a little helper on this challenge? Figure 1: An instance from LifeQA. The image shows a frame from the video, part of the transcriptions, two ques- tions along with the candidate answers, and the correct an- swers in bold. 1. Introduction Video Question Answering (Video QA) is one of the most challenging and crucial problems for artificial intelligence. In this task, we are given a video and must answer natural language questions about its content, such as “What game is the little girl playing?”. Answering these questions requires a rich understanding of the visual and auditory content in the video, as well as the ability to relate this content to natural language concepts. Like many challenging tasks, much of the recent progress on Video QA is due to the introduction of several large-scale datasets, which consist primarily of movies and TV shows (Tapaswi et al., 2016; Rohrbach et al., 2017; Lei et al., 2018). Movies and TV shows provide for countless hours of clean, crisply-edited video and accurately-captioned audio, and are therefore eas- ily adapted into datasets. However, these same features mean that movies and TV are not representative of day-to- day life. Therefore, these datasets cannot be used to evaluate how well models perform when applied to realistic videos of day-to-day life. To address this issue, we introduce Life Question Answer- ing (LifeQA), a Video QA benchmark dataset that consists of videos and questions about day-to-day life. LifeQA is drawn from hand-picked YouTube videos, which depict …

Content truncated to 500 words