Stacking with Auxiliary Features for Visual Question Answering

Proceedings of NAACL-HLT 2018, pages 2217–2226 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Stacking With Auxiliary Features for Visual Question Answering Nazneen Fatema Rajani Department of Computer Science University of Texas at Austin nrajani@cs.utexas.edu Raymond J. Mooney Department of Computer Science University of Texas at Austin mooney@cs.utexas.edu Abstract Visual Question Answering (VQA) is a well- known and challenging task that requires sys- tems to jointly reason about natural language and vision. Deep learning models in vari- ous forms have been the standard for solving VQA. However, some of these VQA models are better at certain types of image-question pairs than other models. Ensembling VQA models intelligently to leverage their diverse expertise is, therefore, advantageous. Stacking With Auxiliary Features (SWAF) is an intelligent ensembling technique which learns to combine the results of multiple mod- els using features of the current problem as context. We propose four categories of aux- iliary features for ensembling for VQA. Three out of the four categories of features can be inferred from an image-question pair and do not require querying the component models. The fourth category of auxiliary features uses model-specific explanations. In this paper, we describe how we use these various categories of auxiliary features to improve performance for VQA. Using SWAF to effectively ensem- ble three recent systems, we obtain a new state-of-the-art. Our work also highlights the advantages of explainable AI models. 1 Introduction Visual Question Answering (VQA), the task of ad- dressing open-ended questions about images (Ma- linowski and Fritz, 2014; Antol et al., 2015), has attracted significant attention in recent years (An- dreas et al., 2016a; Goyal et al., 2016; Agrawal et al., 2016; Teney et al., 2017). Given an im- age and a natural language question about the im- age, the task is to provide an accurate natural lan- guage answer. VQA requires visual and linguis- tic comprehension, language grounding as well as common-sense knowledge. A variety of methods to address these challenges have been developed in recent years (Fukui et al., 2016; Xu and Saenko, 2016; Lu et al., 2016; Chen et al., 2015). The vi- sion component of a typical VQA system extracts visual features using a deep convolutional neural network (CNN), and the linguistic component en- codes the question into a semantic vector using a recurrent neural network (RNN). An answer is then generated conditioned on the visual features and the question vector. Most VQA systems have a single underlying method that optimizes a specific loss function and do not leverage the advantage of using multiple di- verse models. One recent ensembling approach to VQA (Fukui et al., 2016) combined multiple mod- els that use multimodal compact bilinear pool- ing with attention and achieved state-of-the-art ac- curacy on the VQA 2016 challenge. However, their ensemble uses simple softmax averaging to combine outputs from multiple systems. Also, their model is pre-trained on the Visual Genome dataset (Krishna et al., 2017) and they concate- nate learned word embeddings with pre-trained GloVe vectors (Pennington …

Content truncated to 500 words