Co-VQA : Answering by Interactive Sub Question Sequence

Findings of the Association for Computational Linguistics: ACL 2022, pages 2396 - 2408 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Co-VQA : Answering by Interactive Sub Question Sequence Ruonan Wang†, Yuxi Qian†, Fangxiang Feng†, Xiaojie Wang†∗, Huixing Jiang‡ †Beijing University of Posts and Telecommunications ‡Meituan-Dianping Group †{wangrn, qianyuxi, fxfeng, xjwang}@bupt.edu.cn ‡jianghuixing@meituan.com Abstract Most existing approaches to Visual Question Answering (VQA) answer questions directly, however, people usually decompose a complex question into a sequence of simple sub ques- tions and finally obtain the answer to the origi- nal question after answering the sub question sequence(SQS). By simulating the process, this paper proposes a conversation-based VQA (Co- VQA) framework, which consists of three com- ponents: Questioner, Oracle, and Answerer. Questioner raises the sub questions using an extending HRED model, and Oracle answers them one-by-one. An Adaptive Chain Visual Reasoning Model (ACVRM) for Answerer is also proposed, where the question-answer pair is used to update the visual representation sequentially. To perform supervised learning for each model, we introduce a well-designed method to build a SQS for each question on VQA 2.0 and VQA-CP v2 datasets. Experi- mental results show that our method achieves state-of-the-art on VQA-CP v2. Further anal- yses show that SQSs help build direct seman- tic connections between questions and images, provide question-adaptive variable-length rea- soning chains, and with explicit interpretability as well as error traceability. 1 Introduction Visual Question Answering (Agrawal et al., 2015) requires to answer questions about images. It has to process visual and language information simulta- neously, which is a basic ability of advanced agents. Therefore, it has attracted more and more attention (Anderson et al., 2018; Lu et al., 2016; Goyal et al., 2017b; Agrawal et al., 2018). The conventional ap- proach (Agrawal et al., 2015) for Visual Question Answering (VQA) is to encode image and question separately and incorporate the representation of each modality into a joint representation. Recently, with the proposal of Transformer (Vaswani et al., ∗Xiaojie Wang is the corresponding author. Questioner Oracle q1: Is there any snowboard? a1 : yes q3: Is there a man on the far right? a2 : yellow q2: What color is the snowboard? a3 : yes Answerer ans : no Q : Is the snowboard the same color as the jacket of the man on the far right? q4: What color is the man’s jacket? a4 : black Figure 1: An illustrative example. After a se- quence of four sub questions and their answers {(q1,a1),(q2,a2),(q3,a3),(q4,a4)}, its easier to answer the original question. 2017), based on previous dense co-attention mod- els (Kim et al., 2018; Nguyen and Okatani, 2018), some methods (Yu et al., 2019; Gao et al., 2019) fur- ther adopt self-attention mechanism to exploit the fine-grained information in both visual and textual modalities. Meanwhile, to enrich indicative infor- mation about the image contained in the visual rep- resentation, some researchers (Cadène et al., 2019; Li et al., 2019) have explored different methods of relational reasoning to capture the relationship between objects. Though above methods have …

Content truncated to 500 words