Comprehensive Multi-Modal Interactions for Referring Image Segmentation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3427 - 3435 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Comprehensive Multi-Modal Interactions for Referring Image Segmentation Kanishk Jain and Vineet Gandhi CVIT, KCIS, IIIT Hyderabad, India kanishk5991@gmail.com, vgandhi@iiit.ac.in Abstract We investigate Referring Image Segmentation (RIS), which outputs a segmentation map cor- responding to the natural language description. Addressing RIS efficiently requires consider- ing the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interac- tions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmen- tation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hi- erarchy. We present thorough ablation studies and validate our approach’s performance on four benchmark datasets, showing considerable performance gains over the existing state-of- the-art (SOTA) methods. 1 Introduction Traditional computer vision tasks like detection and segmentation have dealt with a pre-defined set of categories, limiting their scalability and practi- cality. Substituting the pre-defined categories with natural language expressions (NLE) is a logical ex- tension to counteract the above problems. Indeed, this is how humans interact with objects in their environment; for example, the phrase “the kid run- ning after the butterfly" requires localizing only the child running after the butterfly and not the other kids. Formally, the task of localizing objects based on NLE is known as Visual Grounding. Existing works either approach the grounding problem by predicting a bounding box around the referred ob- ject or a segmentation mask corresponding to the referred object. We focus on the latter approach, as a segmentation mask can effectively pinpoint Figure 1: Unlike existing methods which model interac- tions in a sequential manner, we synchronously model the Intra-Modal and Inter-Modal interactions across vi- sual and linguistic modalities. Here, Mv and Mt repre- sent Visual and Linguistic Modalities, and {-} represents interactions between them. the exact location and capture the actual shape of the referred object. The task is formally known as Referring Image Segmentation (RIS). RIS requires understanding both visual and lin- guistic modalities at an individual level, specifically word-word and region-region interactions. Addi- tionally, a mutual understanding of both modalities is required to identify the referred object from the linguistic expression and localize it in the image. For instance, to ground a sentence “whatever is on the truck", it is necessary to understand the re- lationship between words as grounding just the individual words will not work. Similarly, region- to-region interactions in visual modality help group semantically similar regions, e.g., all regions be- longing to the truck. Finally, to identify the referent regions, we need to transfer the distinctive informa- tion about the referent from the linguistic modality to the visual modality; this is taken care of by the cross-modal word-region interactions. The current …

Content truncated to 500 words