Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8549 - 8559 December 7-11, 2022 ©2022 Association for Computational Linguistics Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding Keqin Chen1, Richong Zhang1, 2 ∗, Samuel Mensah3, Yongyi Mao4 1SKLSDE, Beihang University, Beijing, China 2Zhongguancun Laboratory, Beijing, China 3Department of Computer Science, University of Sheffield, UK 4 School of Electrical Engineering and Computer Science, University of Ottawa, Canada {chenkq,zhangrc}@act.buaa.edu.cn s.mensah@sheffield.ac.uk, ymao@uottawa.ca Abstract Weakly supervised phrase grounding aims to learn an alignment between phrases in a cap- tion and objects in a corresponding image us- ing only caption-image annotations, i.e., with- out phrase-object annotations. Previous meth- ods typically use a caption-image contrastive loss to indirectly supervise the alignment be- tween phrases and objects, which hinders the maximum use of the intrinsic structure of the multimodal data and leads to unsatisfactory performance. In this work, we directly use the phrase-object contrastive loss in the con- dition that no positive annotation is available in the first place. Specifically, we propose a novel contrastive learning framework based on the expectation-maximization algorithm that adaptively refines the target prediction. Ex- periments on two widely used benchmarks, Flickr30K Entities and RefCOCO+, demon- strate the effectiveness of our framework. We obtain 63.05% top-1 accuracy on Flickr30K Entities and 59.51%/43.46% on RefCOCO+ TestA/TestB, outperforming the previous meth- ods by a large margin, even surpassing a pre- vious SoTA that uses a pre-trained vision- language model. Furthermore, we deliver a theoretical analysis of the effectiveness of our method from the perspective of the maximum likelihood estimate with latent variables. 1 Introduction Phrase grounding aims to localize corresponding objects in an image given a phrase in the image’s caption. It is one of the most fundamental research areas in multimodal learning (Ramachandram and Taylor, 2017). This area has strong applications in other complex visual language tasks, such as vi- sual question answering (Khan et al., 2021), cross- modal retrieval (Chen et al., 2017), etc. ∗Corresponding author pull paired image pull caption-image level phrase-object level another image query caption paired image query caption intrinsic similarity object phrase image caption another image Figure 1: Previous methods neglect the intrinsic similar- ity which can be seen as a soft co-occurrence. Utilizing the intrinsic similarity between negative examples (dark blue circle) and false-positive examples (light blue cir- cle) can push the false-positive examples away from the query phrase (light green triangle). Current methods (Huang et al., 2021; Kamath et al., 2021) have achieved great success but rely heavily on bounding box annotations, which are ex- pensive to acquire. Thus, weakly supervised phrase grounding has recently received increased attention (Liu et al., 2021; Wang et al., 2020; Dou and Peng, 2021) due to the low cost of obtaining image-text pair annotations. It aims to ground phrases us- ing only caption-image annotations, i.e., without phrase-object annotations. The most critical question in weakly super- vised phrase grounding is how to provide suffi- cient phrase-object alignment supervision. Pre- vious methods typically provide oblique phrase- object …

Content truncated to 500 words