Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1217–1229 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective Zijian Zhang∗,1, Chang Shu∗,2,3, Ya Xiao1, Yuan Shen1, Di Zhu1, Jing Xiao2, Youxin Chen2, Jey Han Lau4, Qian Zhang3 and Zheng Lu3 1 Meituan, China 2 Ping An Technology (Shenzhen) Co., Ltd, China 3 University of Nottingham Ningbo, China 4 The University of Melbourne, Australia Abstract Visual-Semantic Embedding (VSE) aims to learn an embedding space where related vi- sual and semantic instances are close to each other. Recent VSE models tend to design complex structures to pool visual and seman- tic features into fixed-length vectors and use hard triplet loss for optimization. However, we find that: (1) combining simple pooling methods is no worse than these sophisticated methods; and (2) only considering the most difficult-to-distinguish negative sample leads to slow convergence and poor Recall@K im- provement. To this end, we propose an adap- tive pooling strategy that allows the model to learn how to aggregate features through a com- bination of simple pooling methods. We also introduce a strategy to dynamically select a group of negative samples to make the opti- mization converge faster and perform better. Experimental results on Flickr30K and MS- COCO demonstrate that a standard VSE us- ing our pooling and optimization strategies out- performs current state-of-the-art systems (at least 1.0% on the metrics of recall) in image- to-text and text-to-image retrieval. Source code of our experiments is available at https: //github.com/96-Zachary/vse_2ad. 1 Introduction Visual Semantic Embedding (VSE) (Frome et al., 2013; Faghri et al., 2018) is a representation learn- ing method that embeds images and texts for effi- cient cross-modal retrieval, and typically has the following steps: (1) the image and text features are first extracted by separate visual and text encoders; (2) these features are then projected into a joint embedding space and pooled to form fixed-length vectors; (3) similarity is computed to measure the distance between image and text representation; and (4) a suitable objective is chosen for optimiza- tion (see Figure 1 for an illustration). Our paper ∗ These authors contributed equally to this work. Feature Extraction Feature Aggregation Similarity Calculation Optimization Target Image Text Inputs Figure 1: Illustration of VSE. focuses on improving the steps of feature aggrega- tion and optimization. For feature aggregation, the most commonly used methods are simple pooling aggregators. MaxPool (Wang et al., 2018) and MeanPool (Reimers and Gurevych, 2019) are designed to de- tect the salient and mean points of features, and K-MaxPool (Kalchbrenner et al., 2014) extracts the mean of top-K features. Some complex aggre- gation techniques have been proposed, e.g. local- importance projection (Gao et al., 2019), sequence- to-sequence encoder (Hu et al., 2019), graph convo- lution network (Li et al., 2019), exponential adap- tive pooling (Stergiou and Poppe, 2021) and self- attention encoder (Wang et al., 2020). However, we found that carefully selected pooling functions can surpass complex methods (see …

Content truncated to 500 words