Multimodal Neural Graph Memory Networks for Visual Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7177–7188 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7177 Multimodal Neural Graph Memory Networks for Visual Question Answering Mahmoud Khademi School of Computing Science Simon Fraser University Burnaby, BC, Canada mkhademi@sfu.ca Abstract We introduce a new neural network architec- ture, Multimodal Neural Graph Memory Net- works (MN-GMN), for visual question answer- ing. The MN-GMN uses graph structure with different region features as node attributes and applies a recently proposed powerful graph neural network model, Graph Network (GN), to reason about objects and their interactions in an image. The input module of the MN-GMN generates a set of visual features plus a set of encoded region-grounded captions (RGCs) for the image. The RGCs capture object at- tributes and their relationships. Two GNs are constructed from the input module using the visual features and encoded RGCs. Each node of the GNs iteratively computes a question- guided contextualized representation of the vi- sual/textual information assigned to it. Then, to combine the information from both GNs, the nodes write the updated representations to an external spatial memory. The final states of the memory cells are fed into an answer mod- ule to predict an answer. Experiments show MN-GMN rivals the state-of-the-art models on Visual7W, VQA-v2.0, and CLEVR datasets. 1 Introduction Visual question answering (VQA) has been recently introduced as a grand challenge for AI. Given an image and a free-form question about it, the VQA task is to produce an accurate natural language answer. VQA has many applications, such as image retrieval and search. This paper proposes a new neural network architecture for VQA based on the recent Graph Network (GN) (Battaglia et al., 2018). The pairwise interactions between various re- gions of an image and spatial context in both hori- zontal and vertical directions are important to an- swer questions about objects and their interactions in the scene context. For example, to answer How many cats are in the picture? (see Figure 1), a Figure 1: An example from Visual Genome (https: //visualgenome.org/). The region-grounded cap- tions provide useful clues to answer questions. For example, to answer Where are the cats?, orange and white cat laying on a wooden bench is informative. model needs to aggregate information from mul- tiple, possibly distant, regions; hence applying a convolutional neural network may not be suf- ficient to perform reasoning over the regions. Our new architecture (see Figure 2), Multimodal Neu- ral Graph Memory Network (MN-GMN), uses a 7178 graph structure to represent pairwise interactions between visual/textual features (nodes) from differ- ent regions of an image. GNs provide a context- aware neural mechanism for computing a feature for each node that represents complex interactions with other nodes. This enables our MN-GMN to an- swer questions that need reasoning about complex arrangements of objects in a scene. Previous approaches such as Memory Networks (MN) (Sukhbaatar et al., 2015) and Dynamic Mem- ory Networks (DMN) (Kumar et al., 2015) com- bined a …

Content truncated to 500 words