Check It Again:Progressive Visual Question Answering via Visual Entailment

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4101–4110 August 1–6, 2021. ©2021 Association for Computational Linguistics 4101 Check It Again: Progressive Visual Question Answering via Visual Entailment Qingyi Si1,2, Zheng Lin1∗, Mingyu Zheng1,2, Peng Fu1, Weiping Wang1 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {siqingyi,linzheng,zhengmingyu,fupeng,wangweiping}@iie.ac.cn Abstract While sophisticated Visual Question Answer- ing models have achieved remarkable success, they tend to answer questions only accord- ing to superficial correlations between ques- tion and answer. Several recent approaches have been developed to address this language priors problem. However, most of them pre- dict the correct answer according to one best output without checking the authenticity of an- swers. Besides, they only explore the inter- action between image and question, ignoring the semantics of candidate answers. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entail- ment. Specifically, we first select the candi- date answers relevant to the question or the im- age, then we rerank the candidate answers by a visual entailment task, which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. Experimental results show the effec- tiveness of our proposed framework, which es- tablishes a new state-of- the-art accuracy on VQA-CP v2 with a 7.55% improvement.1 1 Introduction Visual Question Answering (VQA) task is a multi- modal problem which requires the comprehensive understanding of both visual and textual informa- tion. Presented with an input image and a question, the VQA system tries to determine the correct an- swer in the large prediction space. Recently, some studies (Jabri et al., 2016; Agrawal et al., 2016; Zhang et al., 2016; Goyal et al., 2017) demonstrate that VQA systems suffer from the superficial corre- lation bias (i.e. language priors) caused by acciden- tal correlations between answers and questions. As a result, traditional VQA models always output the ∗Corresponding author: Zheng Lin. 1The code is available at https://github.com/ PhoebusSi/SAR Figure 1: (a) We evaluate the performance of UpDn, LMH, SSL on the VQA-CP v2 test. topN represents the topN accuracy. (b) Visual verification utilizing an- swer semantics. most common answer(Selvaraju et al., 2019) of the input sample’s question category, no matter what image is given. To address this language priors problem, various approaches have been developed. However, through exploring the characteristics of the existing methods, we find that whether the general VQA models such as UpDn(Anderson et al., 2018) and LXMERT(Tan and Bansal, 2019) or models carefully designed for language priors, as LMH(Clark et al., 2019) and SSL(Zhu et al., 2020), yield a non-negligible problem. Both models pre- dict the correct answer according to one best output without checking the authenticity of answers. Be- sides, these models have not made good use of the semantics information of answers that could be helpful for alleviating the language-priors. As presented in Figure 1(a), quite …

Content truncated to 500 words