Language-Conditioned Feature Pyramids for Visual Selection Tasks

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4687–4697 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4687 Language-Conditioned Feature Pyramids for Visual Selection Tasks Taichi Iki1,2 and Akiko Aizawa1,2 1National Institute of Informatics, Chiyoda-ku, Tokyo, Japan 2Graduate University for Advanced Studies, Hayama, Kanagawa, Japan {iki,aizawa}@nii.ac.jp Abstract Referring expression comprehension, which is the ability to locate language to an object in an image, plays an important role in creat- ing common ground. Many models that fuse visual and linguistic features have been pro- posed. However, few models consider the fu- sion of linguistic features with multiple visual features with different sizes of receptive fields, though the proper size of the receptive field of visual features intuitively varies depending on expressions. In this paper, we introduce a neu- ral network architecture that modulates visual features with varying sizes of receptive field by linguistic features. We evaluate our archi- tecture on tasks related to referring expression comprehension in two visual dialogue games. The results show the advantages and broad ap- plicability of our architecture. Source code is available at https://github.com/Alab-NII/lcfp. 1 Introduction Referring expressions are a ubiquitous part of hu- man communication (Krahmer and Van Deemter, 2012) that must be studied in order to create ma- chines that work smoothly with humans. Much ef- fort has been taken to improve methods of creating visual common ground between machines, which have limited means of expression and knowledge about the real world, and humans, from the per- spectives of both referring expression comprehen- sion and generation (Moratz et al., 2002; Tenbrink and Moratz, 2003; Funakoshi et al., 2004, 2005, 2006; Fang et al., 2013). Even now, researchers are exploring possible methods of designing more re- alistic scenarios for applications, such as in visual dialogue games (De Vries et al., 2017; Haber et al., 2019; Udagawa and Aizawa, 2019). Many models have been proposed for referring expression comprehension so far. As image recog- nition matured, Guadarrama et al. (2014) studied Figure 1: Illustration of visual features with different sizes of the receptive fields. Dots represent objects that have color and size as their attributes. Grids in the right three images represent the receptive fields of their vi- sual features. Our architecture fuses linguistic features with each visual feature. object retrieval methods based on category labels predicted by the recognition models. Hu et al. (2016b) extended this approach to broader natu- ral language expression including categories of objects, their attributes, positional configurations, and interactions. In recent years, models that fuse linguistic features with visual features using deep learning have been studied (Hu et al., 2016b,a; An- derson et al., 2018; Deng et al., 2018; Misra et al., 2018; Li et al., 2018; Yang et al., 2019a,b; Liu et al., 2019; Can et al., 2020). When fusing the linguistic features of a spatial referring expression with visual features, the size of the receptive field of visual features 1 is important. Let us take Figure 1 as an example. We can refer to the …

Content truncated to 500 words