Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4812–4822 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4812 Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA Hyounghun Kim Zineng Tang Mohit Bansal UNC Chapel Hill {hyounghk, terran, mbansal}@cs.unc.edu Abstract Videos convey rich information. Dynamic spatio-temporal relationships between peo- ple/objects, and diverse multimodal events are present in a video clip. Hence, it is impor- tant to develop automated models that can ac- curately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the tem- porally relevant information to answer ques- tions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra informa- tion (in explicit textual format to allow eas- ier matching) for answering questions. More- over, our model is also comprised of dual- level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame se- lection problem as a multi-label classification task and introduce two loss functions, In-and- Out Frame Score Margin (IOFSM) and Bal- anced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state- of-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies.1 1 Introduction Recent years have witnessed a paradigm shift in the way we get our information, and a lot of it 1Our code is publicly available at: https://github.com/hyounghk/ VideoQADenseCapFrameGate-ACL2020 is related to watching and listening to videos that are shared in huge amounts via the internet and new high-speed networks. Videos convey a diverse breadth of rich information, such as dynamic spatio- temporal relationships between people/objects, as well as events. Hence, it has become important to develop automated models that can accurately extract such precise multimodal information from videos (Tapaswi et al., 2016; Maharaj et al., 2017; Kim et al., 2017; Jang et al., 2017; Gao et al., 2017; Anne Hendricks et al., 2017; Lei et al., 2018, 2020). Video question answering is a representative AI task through which we can evaluate such abilities of an AI agent to understand, retrieve, and return desired information from given video clips. In this paper, we propose a model that effec- tively integrates multimodal information and lo- cates the relevant frames from diverse, complex video clips such as those from the video+dialogue TVQA dataset (Lei et al., 2018), which contains questions that need both the video and the sub- titles to answer. When given a video clip and a natural language question based on the video, natu- rally, …

Content truncated to 500 words