Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6398–6416 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6398 Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions Arjun R. Akula1, Spandana Gella2, Keze Wang1, Song-Chun Zhu4,5,6, Siva Reddy3 1UCLA Center for Vision, Cognition, Learning, and Autonomy, 2Amazon Alexa AI 3Facebook CIFAR AI Chair, Mila; McGill University 4Beijing Institute for General Artificial Intelligence (BIGAI), 5Tsinghua University, 6Peking University aakula@ucla.edu, sgella@amazon.com, kezewang@gmail.com s.c.zhu@pku.edu.cn, siva.reddy@mila.quebec Abstract Neural module networks (NMN) are a pop- ular approach for grounding visual referring expressions. Prior implementations of NMN use pre-defined and fixed textual inputs in their module instantiation. This necessitates a large number of modules as they lack the ability to share weights and exploit associ- ations between similar textual contexts (e.g. “dark cube on the left” vs. “black cube on the left”). In this work, we address these limitations and evaluate the impact of contex- tual clues in improving the performance of NMN models. First, we address the prob- lem of fixed textual inputs by parameteriz- ing the module arguments. This substan- tially reduce the number of modules in NMN by up to 75% without any loss in perfor- mance. Next we propose a method to con- textualize our parameterized model to enhance the module’s capacity in exploiting the vi- siolinguistic associations. Our model out- performs the state-of-the-art NMN model on CLEVR-Ref+ dataset with +8.1% improve- ment in accuracy on the single-referent test set and +4.3% on the full test set. Addi- tionally, we demonstrate that contextualization provides +11.2% and +1.7% improvements in accuracy over prior NMN models on CLO- SURE and NLVR2. We further evaluate the impact of our contextualization by construct- ing a contrast set for CLEVR-Ref+, which we call CC-Ref+. We significantly outperform the baselines by as much as +10.4% absolute ac- curacy on CC-Ref+, illustrating the generaliza- tion skills of our approach. Our dataset is pub- licly available at https://github.com/ McGill-NLP/contextual-nmn. 1 Introduction Visual referring expression recognition is the task of identifying the object in an image that is referred to by a natural language expression (Kazemzadeh et al., 2014; Mao et al., 2016). It is a fundamental r1: The metallic object that is second one of the thing from right and to the right of [ big cubical metallic thing ] filter shape filter size filter material [cube] [large] [big cubical] Argument Context [metallic] [big cubical metallic thing] [cubical] e1 e1 r1 Figure 1: An example from the CLEVR-Ref+ dataset. In addition to passing textual inputs (arguments) cubical, large and metallic to neural modules, we also provide them with the relevant neighborhood of arguments as context (highlighted in blue). language-to-vision matching problem and has sev- eral downstream applications such as question an- swering, robot navigation, and image retrieval (Zhu et al., 2016; Qi et al., 2020; Young et al., 2014; Yang et al., 2016; Tu et al., 2014; Qi et al., 2015; Liu et al., 2016; Akula and Zhu, …

Content truncated to 500 words