Towards Knowledge-Augmented Visual Question Answering

Proceedings of the 28th International Conference on Computational Linguistics, pages 1863–1873 Barcelona, Spain (Online), December 8-13, 2020 1863 Towards Knowledge-Augmented Visual Question Answering Maryam Ziaeefard McGill University Montr´eal, Canada Freddy L´ecu´e Inria, Sophia Antipolis, France Thales, Montr´eal, Canada Abstract Visual Question Answering (VQA) remains algorithmically challenging while it is effortless for humans. Humans combine visual observations with general and commonsense knowledge to answer a question about a given image. In this paper, we address the problem of incorporating general knowledge into VQA models while leveraging the visual information. We propose a model that captures the interactions between objects in a visual scene and entities in an exter- nal knowledge source. Our model is a graph-based approach that combines scene graphs with concept graphs, which learns a question-adaptive graph representation of related knowledge in- stances. We use Graph Attention Networks to set higher importance to key knowledge instances that are mostly relevant to each question. We exploit ConceptNet as the source of general knowl- edge and evaluate the performance of our model on the challenging OK-VQA dataset. Our code will be available at https://github.com/ZiaMaryam/KVQA 1 Introduction The task of Visual Question Answering (VQA) (Antol et al., 2015) was introduced to bridge the gap between natural language processing and image understanding applications. Most VQA methods focus on the visual aspect of the VQA task and predict the answer by combining the question and image representations. However, visual-based approaches are practical when no insights beyond the visual content is required. Incorporating external knowledge into VQA models combines visual observations with external knowledge (Garderes et al., 2020). Organizing the external knowledge and storing them in a struc- tured database, such as a Knowledge Bases (KB), have become important resources for representing the general knowledge. A typical KB could be represented as a graph, which is usually a collection of triples also known as facts. The triples specify that two entities (nodes) are connected by a particular relation (edge), e.g., (Shakespeare, writerOf, Hamlet) where Shakespeare and Hamlet are en- tities and writerOf is the relation between the entities. Example of such graph structures are YAGO (Suchanek et al., 2007), DBpedia (Auer et al., 2007), NELL (Carlson et al., 2010), Freebase (Bollacker et al., 2008), and the Google Knowledge Graph (Steiner et al., 2012). In recent years, a significant amount of research has been devoted to visual-based VQA while external knowledge remains unavailable. To address this challenge, recent work (Wang et al., 2017; Wang et al., 2018; Shah et al., 2019) generated new datasets for the purpose of evaluating the performance of VQA algorithms capable of answering higher knowledge level. We use OK-VQA dataset (Marino et al., 2019) for our experiments since it requires handling outside knowledge. In addition to understanding the question and the image, the model needs to learn what knowledge is necessary to answer the OK-VQA questions since no ground-truth facts are provided with the dataset. In this work, our main motivation is to develop a technique to integrate general knowledge while lever- …

Content truncated to 500 words