MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4648–4660 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4648 MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering Aisha Urooj Khan Amir Mazaheri Niels Da Vitoria Lobo Mubarak Shah Center for Research in Computer Vision, University of Central Florida aishaurooj@gmail.com, amirmazaheri@knights.ucf.edu shah@crcv.ucf.edu, niels@cs.ucf.edu Abstract We present MMFT-BERT (MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer- based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator’s judgment. This set of questions helps us to study the model’s behavior and the challenges TVQA poses to prevent the achievement of super human performance. Extensive experiments show the effectiveness and superiority of our method1. 1 Introduction In the real world, acquiring knowledge requires processing multiple information sources such as visual, sound, and natural language individually and collectively. As humans, we can capture ex- perience from each of these sources (like an iso- lated sound); however, we acquire the maximum knowledge when exposed to all sources concur- rently. Thus, it is crucial for an ideal Artificial Intelligence (AI) system to process modalities indi- vidually and jointly. One of the ways to understand and communicate with the world around us is by observing the environment and using language (dia- logue) to interact with it (Lei et al., 2018). A smart 1Code will be available at https://github.com/ aurooj/MMFT-BERT Figure 1: MultiModal Fusion Transformer (MMFT): We treat input modalities as a sequence. [FUSE] is a trainable vector; hq0 j,hv0 j, and hs0 j are fixed-length features aggregated over question-answer (QA) pairs, visual concepts, and subtitles. Using a transformer encoder block, [FUSE] attends all source vectors and assigns weights based on the importance of each in- put source. Training end to end for VQA enables the MMFT module to learn to aggregate input sources w.r.t. the nature of the question. For illustration purposes, we show that for a single head, MMFT collects more knowledge from the visual source hv0 j (green colored) than from the QA and subtitles. Best viewed in color. system, therefore, should be able to process visual information to extract meaningful knowledge as well as be able to use that knowledge to tell us what is happening. The story is incomplete if we isolate the visual domain from language. Now that advancements in both computer vision and natural language processing are substantial, solving prob- lems demanding multimodal understanding (their fusion) is the next step. Answering questions about what can be seen and heard lies somewhere along this direction of investigation. In …

Content truncated to 500 words