KAT: A Knowledge Augmented Transformer for Vision-and-Language

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 956 - 968 July 10-15, 2022 ©2022 Association for Computational Linguistics KAT: A Knowledge Augmented Transformer for Vision-and-Language Liangke Gui§‡ Borui Wang†‡ Qiuyuan Huang‡ Alex Hauptmann§ Yonatan Bisk§‡ Jianfeng Gao‡ §Carnegie Mellon University †Yale University ‡Microsoft Research {liangkeg, alex, ybisk}@cs.cmu.edu borui.wang@yale.edu, {qihua, jfgao}@microsoft.com Abstract The primary focus of recent work with large- scale transformers has been on optimizing the amount of information packed into the model’s parameters. In this work, we ask a complemen- tary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer predic- tion, but leave open questions about the qual- ity and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be in- tegrated. To address these challenges, we pro- pose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the- art result (+6% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, ex- plicit knowledge integration improves inter- pretability of model predictions in our analysis. Code and pre-trained models are released at https://github.com/guilk/KAT. 1 Introduction There has been a revival of interest in knowledge- intensive tasks which require an external knowl- edge source for humans to perform. Many applica- tions in real-world scenarios, such as autonomous AI agents, need to seamlessly integrate implicit (i.e., commonsense) and explicit knowledge (e.g., Wikidata) to answer questions. In this work, we investigate how to effectively integrate implicit and explicit knowledge for reasoning. Tasks like Out- side Knowledge Visual Question Answering (OK- VQA) (Marino et al., 2019) require that models use knowledge not present in the input to answer ques- Work done when Liangke and Borui interned at Microsoft Research. Figure 1: Examples of knowledge-based VQA that re- quires external knowledge. Success on this task requires not only visual recognition, but also logical reasoning to incorporate external knowledge about the world. tions, making it an ideal test bed for investigating this implicit-explicit knowledge trade-off. Consider the examples from OK-VQA shown in Figure 1. To answer the question in the left exam- ple, the system needs to both ground organism to bird through explicit knowledge and then apply the implicit knowledge birds evolved from reptiles to answer the question. Similarly for the question in the right example, the system needs to recognize boats and harbor and requires the implicit knowl- edge anchors are used to stop boats from moving. A key challenge here is to accurately link image content to abstract external knowledge. There have been a number of recent developments demonstrat- ing the feasibility of incorporating external knowl- edge into Question Answering models (Wang et al., 2017b; Li et al., 2020b; Marino et al., 2021; Wu et al., 2022; Garderes et al., …

Content truncated to 500 words