Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3285–3292, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3285 Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering Zujie Liang, Weitao Jiang, Haifeng Hu, Jiaying Zhu School of Electronics and Information Technology, Sun Yat-sen University {liangzj9,jiangwt5,zhujy53}@mail2.sysu.edu.cn, huhaif@mail.sysu.edu.cn Abstract In the task of Visual Question Answering (VQA), most state-of-the-art models tend to learn spurious correlations in the training set and achieve poor performance in out-of- distribution test data. Some methods of gen- erating counterfactual samples have been pro- posed to alleviate this problem. However, the counterfactual samples generated by most pre- vious methods are simply added to the train- ing data for augmentation and are not fully uti- lized. Therefore, we introduce a novel self- supervised contrastive learning mechanism to learn the relationship between original sam- ples, factual samples and counterfactual sam- ples. With the better cross-modal joint embed- dings learned from the auxiliary training objec- tive, the reasoning capability and robustness of the VQA model are boosted significantly. We evaluate the effectiveness of our method by surpassing current state-of-the-art models on the VQA-CP dataset, a diagnostic benchmark for assessing the VQA model’s robustness. 1 Introduction To develop human-like visual and language under- standing of AI, the task of answering a question about the given visual content has been proposed, i.e., Visual Question Answering (VQA) (Antol et al., 2015). Although the current state-of-the-art methods (Fukui et al., 2016; Anderson et al., 2018; Cadene et al., 2019a) can achieve good results on the VQA benchmarks such as VQA v2 (Goyal et al., 2017), recent researches (Agrawal et al., 2016; Kafle and Kanan, 2017; Agrawal et al., 2018) have found that these methods tend to explore superficial correlations in the training set and perform poorly when transferred to real world setting. Specifically, given a question “What color is the banana?”, the models prefer to take the shortcut and “assume” that the answer should be “yellow” since it is the most common answer in the training set, rather Is this man holding a cat? 𝑸 Is this man holding a [MASK]? Is [MASK] [MASK] [MASK] [MASK] cat? 𝑰 Yes. Yes. NOT Yes. 𝒂 𝑰+ 𝑰− 𝑸+ 𝑸− 𝒂+ 𝒂− Figure 1: An informal examples of original sam- ple (I, Q), factual sample (I+, Q+) and counterfactual sample (I−, Q−) generated by the counterfactual sam- ple synthesizing algorithm (Chen et al., 2020). than be grounded on the image. To overcome the language bias problems in VQA, (Agrawal et al., 2018) have proposed a dataset named VQA-CP, where the answer distribution of the training set differs from the test set vastly. The performance of most current state-of-the-art models (Andreas et al., 2016; Teney et al., 2018; Shrestha et al., 2019) drop significantly on the VQA-CP due to the language bias. Hence, it has become the standard out-of-distribution benchmark for VQA. A successful robust and unbiased VQA system is supposed to be able to deduce the right answer from the right …

Content truncated to 500 words