Dual Capsule Attention Mask Network with Mutual Learning for Visual Question Answering

Proceedings of the 29th International Conference on Computational Linguistics, pages 5678–5688 October 12–17, 2022. 5678 Dual Capsule Attention Mask Network with Mutual Learning for Visual Question Answering Weidong Tian1,3, Haodong Li1,3, Zhong-Qiu Zhao1,2,3,4,∗ 1College of Computer and Information, Hefei University of Technology 2Guangxi Academy of Science 3Intelligent Manufacturing Institute of HFUT 4Intelligent Interconnected Systems Laboratory of Anhui Province, Hefei University of Technology wdtian@hfut.edu.cn,lhd_hfut@163.com,z.zhao@hfut.edu.cn Abstract A Visual Question Answering (VQA) model processes images and questions simultaneously with rich semantic information. The attention mechanism can highlight fine-grained features with critical information, thus ensuring that feature extraction emphasizes the objects re- lated to the questions. However, unattended coarse-grained information is also essential for questions involving global elements. We be- lieve that global coarse-grained information and local fine-grained information can com- plement each other to provide richer compre- hensive information. In this paper, we propose a dual capsule attention mask network with mutual learning for VQA. Specifically, it con- tains two branches processing coarse-grained features and fine-grained features, respectively. We also design a novel stackable dual capsule attention module to fuse features and locate evidence. The two branches are combined to make final predictions for VQA. Experimental results show that our method outperforms the baselines in terms of VQA performance and interpretability and achieves new SOTA perfor- mance on the VQA-v2 dataset. 1 Introduction In recent years, visual question answering (VQA) has received extensive research attention in the fields of computer vision and multimedia comput- ing. The goal of VQA is to answer questions re- lated to the content of images correctly (Antol et al., 2015). It has a wide range of practice applications, such as helping people with visual impairment and human-computer Q&A. In the early stage, most VQA models ex- tract features from images and questions indepen- dently (Malinowski et al., 2015; Gao et al., 2015; Ren et al., 2015). These methods fail to capture the fine-grained key features and include much un- necessary information. Afterward, the attention ∗Corresponding author. Q:What is on the little girls head ? A:helmet Q:Could this be a multi-purpose room? A:yes Figure 1: Samples in the VQA-v2 dataset. (Left): The fine-grained features with attention have the critical in- formation required for the answer inference, which can help the model generate the correct answer by elimi- nating the interference of irrelevant factors. (Right): Unattended coarse-grained features have richer seman- tic information, which can help the answer inference when the attention mechanism is of limited use. mechanism becomes popular and is introduced in many fields (Lu et al., 2016; Cai and Hu, 2020). The VQA models with the attention mechanism ex- tract critical information from one modality guided by another modality (Lu et al., 2016; Anderson et al., 2018; Yu et al., 2019). Consider the question related to Figure 1 (Left) “What is on the little girl’s head?” The attention method needs to encourage the model to focus on the "girls head" in the ques- tion and related regions in the image to produce the correct answer as “helmet.” …

Content truncated to 500 words