Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances

Proceedings of the 29th International Conference on Computational Linguistics, pages 5721–5729 October 12–17, 2022. 5721 Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances Yike Wu1 Yu Zhao1 Shiwan Zhao Ying Zhang1∗ Xiaojie Yuan1 Guoqing Zhao2 Ning Jiang2 1 Nankai University, Tianjin, China 2 Mashang Consumer Finance Co, Ltd wuyike@nankai.edu.cn, zhaoyu@dbis.nankai.edu.cn zhaosw@gmail.com, {yingzhang,yuanxj}@nankai.edu.cn Abstract Despite the great progress of Visual Question Answering (VQA), current VQA models heav- ily rely on the superficial correlation between the question type and its corresponding fre- quent answers (i.e., language priors) to make predictions, without really understanding the input. In this work, we define the training in- stances with the same question type but differ- ent answers as superficially similar instances, and attribute the language priors to the con- fusion of VQA model on such instances. To solve this problem, we propose a novel training framework that explicitly encourages the VQA model to distinguish between the superficially similar instances. Specifically, for each training instance, we first construct a set that contains its superficially similar counterparts. Then we exploit the proposed distinguishing module to increase the distance between the instance and its counterparts in the answer space. In this way, the VQA model is forced to further fo- cus on the other parts of the input beyond the question type, which helps to overcome the lan- guage priors. Experimental results show that our method achieves the state-of-the-art perfor- mance on VQA-CP v2. Codes are available at Distinguishing-VQA. 1 Introduction Recent years have witnessed great progress in VQA based on deep learning. However, some researchers reveal that most existing VQA models heavily rely on the superficial correlation between the ques- tion type and its corresponding frequent answers to make predictions, instead of really understand- ing the input (Agrawal et al., 2016, 2018; Goyal et al., 2017; Zhang et al., 2016). For example, once a VQA model detects that a question begins with “how many”, it tends to blindly output the most common answer “2” to the “how many” questions in the training data without looking at other parts of the input. ∗Corresponding author. How many animals are in the picture? GT: 2 How many buses are there? GT: 1 (a) An example of superficially similar instances 3 2 1 0 1 2 3 4 5 6 4 2 0 2 4 6 8 10 SAN GT: 1 GT: 2 2 0 2 4 6 8 10 4 2 0 2 4 UpDn GT: 1 GT: 2 (b) Superficially similar instances in the answer space Figure 1: (a) displays an example of superficially similar instances, which have the same question type (“How many”) but different ground-truth answers (“1” and “2”). (b) visualizes the projection of such instances in the answer space obtained by SAN (Yang et al., 2016) and UpDn (Anderson et al., 2018) respectively. We observe that both models could not distinguish well between the mixed superficially similar instances. We argue that the language priors arise from that the VQA model only captures the …

Content truncated to 500 words