Overcoming Language Priors with Counterfactual Inference for Visual Question Answering

Overcoming Language Priors with Counterfactual Inference for Visual Question Answering Zhibo Ren, Huizhen Wang∗, Muhua Zhu, Yichao Wang, Tong Xiao, Jingbo Zhu NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China rzb1998@qq.com wanghuizhen@mail.neu.edu.cn Abstract Recent years have seen a lot of efforts in attacking the issue of language priors in the field of Visual Question Answering (VQA). Among the extensive efforts, causal inference is regarded as a promising direction to mitigate language bias by weakening the direct causal effect of questions on answers. In this paper, we follow the same direction and attack the issue of language priors by incorporating counterfactual data. Moreover, we propose a two-stage training strategy which is deemed to make better use of counterfactual data. Experiments on the widely used bench- mark VQA-CP v2 demonstrate the effectiveness of the proposed approach, which improves the baseline by 21.21% and outperforms most of the previous systems. 1 Introduction As an AI-complete task to answer questions about visual content, Visual Question Answering (VQA) has seen surging interest in recent years. The task is thought to be extremely challenging since a VQA system requires the capability of visual and language understanding and the capability of multi-modal reasoning. Recent researches in this field have paid increasing attention to the issue of language priors, aka language bias (Agrawal et al., 2018). The issue of language priors is caused by spurious correlation between the question pattern and the answer. See the example in Figure 1, “yellow” is the most likely answer to the question “what color are the bananas” in the training data. So a simple solution to answering the question is to give the answer “yellow” with no reference to visual content. Such a short cut can achieve an accuracy of 54.5% for the question. To overcome language priors in VQA, previous works generally resort to data augmentation. In this direction, visual and textual explanations can be used as the data for augmentation (Das et al., 2017; Park et al., 2018). Besides, counterfactual training samples are also regarded as a valuable source for the purpose (Chen et al., 2020; Zhu et al., 2020; Gokhale et al., 2020; Liang et al., 2020). In the direction of causal effect for VQA, more recent work is counterfactual VQA that focuses on the inference instead of training phase (Niu et al., 2021), though, we still think of counterfactual data augmentation as an efficient and effective way to solve the issue of language priors. So in this paper we first design novel causal graphs specifically for the task of VQA, and then use the causal graphs to guide the generation of counterfactual data. Finally, to make better use of counterfactual data, we propose a two-stage training strategy. We evaluate the proposed approach on the widely used benchmark VQA-CP v2. Extensive experiments demonstrate the effectiveness of the approach, which improves over the baseline by 21.21% and outperforms most of previous systems. Moreover, to evaluate the generalization ability of the ap- proach, we also experiment with …

Content truncated to 500 words