Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6167–6177 August 1–6, 2021. ©2021 Association for Computational Linguistics 6167 Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering Ahjeong Seo1, Gi-Cheon Kang1,2, Joonhan Park3,∗, Byoung-Tak Zhang1,2 1Seoul National University 2AI Institute for Seoul National University (AIIS) 3 Hanyang University {ajseo,gckang,jhpark,btzhang}@bi.snu.ac.kr Abstract Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1) understand the intention of various questions, (2) capturing various el- ements of the input video (e.g., object, ac- tion, causality), and (3) cross-modal ground- ing between language and vision information. We propose Motion-Appearance Synergistic Networks (MASN), which embed two cross- modal features grounded on motion and ap- pearance information and selectively utilize them depending on the question’s intentions. MASN consists of a motion module, an ap- pearance module, and a motion-appearance fu- sion module. The motion module computes the action-oriented cross-modal joint represen- tations, while the appearance module focuses on the appearance aspect of the input video. Finally, the motion-appearance fusion module takes each output of the motion module and the appearance module as input, and performs question-guided fusion. As a result, MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets. We also conduct qualitative analysis by visual- izing the inference results of MASN. The code is available at https://github.com/ ahjeongseo/MASN-pytorch. 1 Introduction Recently, research in natural language processing and computer vision has made significant progress in artificial intelligence (AI). Thanks to this, vision- language tasks such as image captioning (Xu et al., 2015), visual question answering (VQA) (Antol et al., 2015; Goyal et al., 2017), and visual com- monsense reasoning (VCR) (Zellers et al., 2019) have been introduced to the research community, ∗ Work done during an internship at AI Institute for Seoul National University (AIIS). along with some benchmark datasets. In particular, video question answering (video QA) tasks (Xu et al., 2016; Jang et al., 2017; Lei et al., 2018; Yu et al., 2019; Choi et al., 2020) have been proposed with the goal of reasoning over higher-level vision- language interactions. In contrast to QA tasks based on static images, the questions presented in the video QA dataset vary from frame-level questions regarding the appearance of objects (e.g., what is the color of the hat?) to questions regarding ac- tion and causality (e.g., what does the man do after opening a door?). There are three crucial challenges in video QA: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, and causality), and (3) cross- modal grounding between language and vision in- formation. To tackle these challenges, previous studies (Li et al., 2019; Jiang et al., 2020; Huang et al., 2020) have mainly explored this task by jointly embedding the features from the pre-trained word embedding model (Pennington et al., 2014) and the object detection models (He …

Content truncated to 500 words