Contrastive Visual and Language Learning for Visual Relationship Detection

Contrastive Visual and Language Learning for Visual Relationship Detection Thanh Tran and Ma¨elic Neau and Paulo E. Santos and David Powers {tran0553, neau001, paulo.santos, david.powers}@flinders.edu.au College of Science and Engineering Flinders University 1284 South Rd, Clovelly Park, SA5042 Australia Abstract Visual Relationship Detection (VRD) aims to understand real-world objects’ interactions by grounding visual concepts to compositional vi- sual relation triples, written in the form of (sub- ject, predicate, object). Previous work explored the use of contrastive learning to implicitly pre- dict predicates (representing relations) from the relevant image regions. However, these mod- els often directly leverage in-distribution spa- tial and language co-occurrences biases during training, preventing the models from general- izing to out-of-distribution compositions. In this work, we examined whether contrastive vi- sion and language models, pre-trained on large- scale external image and text datasets, can as- sist the detection of compositional visual rela- tions. To this end, we propose a contrastive fine-tuning approach for the VRD task. The results obtained from this investigation show that larger models yield better performance when compared with their smaller counterparts, while models pre-trained on larger datasets do not necessarily present the best performance. 1 Introduction Understanding the visual world is essential for many modern computer vision tasks, including vi- sual question answering (VQA) (Agrawal et al., 2016), image captioning (Hossain et al., 2019), and human-robot interaction (Goodrich and Schultz, 2007). Given the complexity of real world scenes, low-level object classification and recognition tasks are insufficient to solve these problems. Instead, higher-level visual understanding and reasoning tasks are often required. Visual Relationship Detec- tion (VRD) aims to facilitate such understanding by bridging the gap between low-level visual in- formation and high-level symbolic visual relation. Previous work (Lu et al., 2016) on VRD empha- sized the use of spatial priors in the form of overlap- ping bounding boxes and language co-occurrence bias during training to achieve higher benchmark results. However, such explicit incorporation of pri- ors during training often leads to biases that might not transfer to new visual relationship (Zhu et al., 2022). To address these issues, current research lever- age external linguistic commonsense knowledge from structured knowledge bases (Zareian et al., 2020) or unstructured natural language corpora (Ye and Kovashka, 2021) by distilling the information down into low-dimensional distributed vector em- beddings. These embeddings indirectly benefit the visual relationship detection task as they preserve the relevant topological structure from the knowl- edge base or the linguistic contextual information from raw language corpora. In this work, we focus on the later and examine whether contrastive em- bedding models learned from the abundant amount of unstructured image and text pairs from the web (see Table 1) can contribute to the VRD task. To this end, we examined the promise of robust- ness in the large pre-trained image and text deep neural encoders (Radford et al., 2021), focusing on learning joint visual and language embeddings for VRD. The present paper investigates the applica- tion of contrastive learning in this context. Con- trastive learning aims to learn …

Content truncated to 500 words