Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457–468, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding Akira Fukui*1,2 Dong Huk Park*1 Daylen Yang*1 Anna Rohrbach*1,3 Trevor Darrell1 Marcus Rohrbach1 1UC Berkeley EECS, CA, United States 2Sony Corp., Tokyo, Japan 3Max Planck Institute for Informatics, Saarbr¨ucken, Germany Abstract Modeling textual or visual information with vector representations trained from large lan- guage or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combin- ing these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as con- catenation of the visual and textual represen- tations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multi- modal features. We extensively evaluate MCB on the visual question answering and ground- ing tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architec- ture which uses MCB twice, once for predict- ing attention over spatial features and again to combine the attended representation with the question representation. This model out- performs the state-of-the-art on the Visual7W dataset and the VQA challenge. 1 Introduction Representation learning for text and images has been extensively studied in recent years. Recurrent neural networks (RNNs) are often used to represent sen- tences or phrases (Sutskever et al., 2014; Kiros et al., * indicates equal contribution CNN WE, LSTM What are all the people doing? Ψ Ψ FFT FFT-1 Convolution ⨀ FFT “flying kites” Classifier Multimodal Compact Bilinear Count Sketch Signed Sqrt L2 Norm Figure 1: Multimodal Compact Bilinear Pooling for visual question answering. 2015), and convolutional neural networks (CNNs) have shown to work best to represent images (Don- ahue et al., 2013; He et al., 2015). For tasks such as visual question answering (VQA) and visual ground- ing, most approaches require joining the represen- tation of both modalities. For combining the two vector representations (multimodal pooling), current approaches in VQA or grounding rely on concatenat- ing vectors or applying element-wise sum or product. While this generates a joint representation, it might not be expressive enough to fully capture the complex associations between the two different modalities. In this paper, we propose to rely on Multimodal Compact Bilinear pooling (MCB) to get a joint repre- sentation. Bilinear pooling computes the outer prod- uct between two vectors, which allows, in contrast to element-wise product, a multiplicative interaction between all elements of both vectors. Bilinear pool- ing models (Tenenbaum and Freeman, 2000) have recently been shown to be beneficial for fine-grained classification for vision only tasks (Lin et al., 2015). However, given their high dimensionality (n2), bi- linear pooling has so far not been widely …

Content truncated to 500 words