RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4800 - 4810 December 7-11, 2022 ©2022 Association for Computational Linguistics RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning Yi Zhu1∗, Zhaoqing Zhu2∗, Bingqian Lin3, Xiaodan Liang3, Feng Zhao2†, Jianzhuang Liu1 1Huawei Noah’s Ark Lab, 2University of Science and Technology of China, 3Sun Yat-sen University zhu.yee@outlook.com, zhaoqingzhu@mail.ustc.edu.cn, bingqianlin@126.com, xdliang328@gmail.com, fzhao956@ustc.edu.cn, liu.jianzhuang@huawei.com Abstract Conventional visual relationship detection mod- els only use the numeric ids of relation labels for training, but ignore the semantic correlation between the labels, which leads to severe train- ing biases and harms the generalization ability of representations. In this paper, we introduce compact language information of relation la- bels for regularizing the representation learn- ing of visual relations. Specifically, we propose a simple yet effective visual Relationship pre- diction framework that transfers natural lan- guage knowledge learned from Contrastive Language-Image Pre-training (CLIP) models to enhance the relationship prediction, termed as RelCLIP. Benefiting from the powerful visual-semantic alignment ability of CLIP at image level, we introduce a novel Relational Contrastive Learning (RCL) approach that ex- plores relation-level visual-semantic alignment via learning to match cross-modal relational embeddings. By collaboratively learning the semantic coherence and discrepancy from rela- tion triplets, the model can generate more dis- criminative and robust representations. Exper- imental results on the Visual Genome dataset show that RelCLIP achieves significant im- provements over strong baselines under full (providing accurate labels) and distant supervi- sion (providing noise labels), demonstrating its powerful generalization ability in learning rela- tionship representations. Code will be available at https://gitee.com/mindspore/models/ tree/master/research/cv/RelCLIP. 1 Introduction Visual relationship detection needs to predict the re- lation label between a pair of localized objects (e.g. “man carrying bag”). Based on such relationships we can construct a structural representation (i.e., scene graph) that regards the visual concepts within a scene as a whole and could benefit many down- stream reasoning tasks, such as image retrieval (Qi ∗ Equal contribution. † Corresponding author. ridding carrying visual embedding language embedding visual embedding space language embedding space man riding subject-predicate predicate-object carrying surfboard carrying skateboard Relational Contrastive Learning ridding carrying Figure 1: Overview of RelCLIP which adapts Con- trastive Language-Image Pretraining (CLIP) models to enhance the visual-semantic alignment for relationship learning via a novel Relational Contrastive Learning (RCL) approach. RCL matches the cross-modal re- lationship embeddings in both subject-predicate and predicate-object levels and improves relationship dis- crimination with the help of language information of relation labels. et al., 2017; Johnson et al., 2015), vision question answering (Antol et al., 2015), and visual common- sense reasoning (Zellers et al., 2019). Existing visual relationship detection models (Yao et al., 2021; Guo et al., 2021; Chen et al., 2019; Zellers et al., 2018) learn relation represen- tations based on the visual information of object pairs under the supervision of numeric ids of rela- tion labels. The learned representations often suffer from the highly changeful visual appearances of the instances from the same relation category as well as the fine-detailed …

Content truncated to 500 words