Combo of Thinking and Observing for Outside-Knowledge VQA

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10959–10975 July 9-14, 2023 ©2023 Association for Computational Linguistics Combo of Thinking and Observing for Outside-Knowledge VQA Qingyi Si1,2, Yuchen Mo3, Zheng Lin1,2∗, Huishan Ji1,2, Weiping Wang1 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3ByteDance AI Lab, Beijing, China {siqingyi,linzheng,jihuishan,wangweiping}@iie.ac.cn Abstract Outside-knowledge visual question answering is a challenging task that requires both the ac- quisition and the use of open-ended real-world knowledge. Some existing solutions draw ex- ternal knowledge into the cross-modality space which overlooks the much vaster textual knowl- edge in natural-language space, while oth- ers transform the image into a text that fur- ther fuses with the textual knowledge into the natural-language space and completely aban- dons the use of visual features. In this paper, we are inspired to constrain the cross-modality space into the same space of natural-language space which makes the visual features pre- served directly, and the model still benefits from the vast knowledge in natural-language space. To this end, we propose a novel frame- work consisting of a multimodal encoder, a textual encoder and an answer decoder. Such structure allows us to introduce more types of knowledge including explicit and implicit mul- timodal and textual knowledge. Extensive ex- periments validate the superiority of the pro- posed method which outperforms the state-of- the-art by 6.17% accuracy. We also conduct comprehensive ablations of each component, and systematically study the roles of varying types of knowledge. Codes and knowledge data can be found at https://github.com/ PhoebusSi/Thinking-while-Observing.1 1 Introduction Conventional visual question answering (VQA) (Antol et al., 2015) tasks require models to answer questions based on image content. Such tasks have been thoroughly studied (Guo et al., 2021; Jiang et al., 2020; Li et al., 2020b) on conventional VQA datasets VQAv2 (Goyal et al., 2017). However, real-world questions often rely on a certain amount of knowledge beyond images. Therefore, Knowl- edge Base Question Answering (KB-VQA) tasks ∗Corresponding author: Zheng Lin. 1Joint work with ByteDance AI Lab. Image Question Cross-modality Space (a) Conventional VQA Paradigm multimodal encoder classifier Image Question (b) Language-centric Paradigm Visual Context Knowledge textual encoder textual decoder Natural-Language Space Knowledge Image Question textual encoder textual decoder multimodal encoder Combo of Cross-modality Space and Natural-Language Space Visual Context (c) The Proposed Paradigm Figure 1: Comparison with previous paradigms. Orange lines indicate processes involving cross-modality space. (a) The conventional VQA paradigm fuses image and question text into the cross-modality space, and then predicts answers in a close-set classification manner. (b) Language-centric paradigm applies captioning and tagging tools to describe the visual context, and aban- dons the visual features to convert the VQA task into an open-ended generative QA task. (c) The proposed paradigm intends to constrain the cross-modality space into the same space as natural-language space so that models can directly decode both text and multimodal embeddings. (Cao et al., 2021; Wang et al., 2015, 2017; …

Content truncated to 500 words