TVQA+: Spatio-Temporal Grounding for Video Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8211–8225 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8211 TVQA+: Spatio-Temporal Grounding for Video Question Answering Jie Lei Licheng Yu Tamara L. Berg Department of Computer ScienceMohit Bansal University of North Carolina at Chapel Hill {jielei, licheng, tlberg, mbansal}@cs.unc.edu Abstract We present the task of Spatio-Temporal Video Question Answering, which requires intelli- gent systems to simultaneously retrieve rel- evant moments and detect referenced visual concepts (people and objects) to answer nat- ural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to vi- sual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified frame- work that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the ques- tion answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal at- tention visualizations.1 1 Introduction We have witnessed great progress in recent years on image-based visual question answering (QA) tasks (Antol et al., 2015; Yu et al., 2015; Zhu et al., 2016b). One key to this success has been spatial at- tention (Anderson et al., 2018; Shih et al., 2016; Lu et al., 2016), where neural models learn to attend to relevant regions for predicting the correct answer. Compared to image-based QA, there has been less progress on the performance of video-based QA tasks. One possible reason is that attention tech- niques are hard to generalize to the temporal na- ture of videos. Moreover, due to the high cost of annotation, most existing video QA datasets only contain QA pairs, without providing labels for the 1Dataset and code are publicly available: http: //tvqa.cs.unc.edu, https://github.com/ jayleicn/TVQAplus Question: What is Sheldon holding when he is talking to Howard about the sword? Correct Answer: A computer. 00:02.314 → 00:06.732 Howard: Sheldon, he’s got Raj. Use your sleep spell. Sheldon! Sheldon! 00:06.902 → 00:10.992 Sheldon: I’ve got the Sword of Azeroth. Question: Who is talking to Howard when he is in the kitchen upset? Correct Answer: Raj is talking to Howard. 00:17.982 → 00:20.532 Howard: That's really stupid advice. 00:20.534 → 00:22.364 Raj: You know that hurts my feelings. Figure 1: Samples from TVQA+. Questions and cor- rect answers are temporally localized to clips, and spa- tially localized to frame-level bounding boxes. Colors indicate corresponding box-object pairs. Subtitles are shown in dashed blocks. Wrong answers are omitted. key clips or regions needed to answer the question. Inspired by previous work on grounded image and video captioning (Lu et al., 2018; Zhou et al., 2019), we propose methods that explicitly localize video clips as well as spatial regions for answering video- based questions. Such methods are useful in many scenarios, such as natural language guided …

Content truncated to 500 words