ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14507–14525 July 9-14, 2023 ©2023 Association for Computational Linguistics ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning Xiao Xu1, 3∗, Bei Li2, 3, Chenfei Wu3, Shao-Yen Tseng4, Anahita Bhiwandiwalla4, Shachar Rosenman4, Vasudev Lal4, Wanxiang Che1†, Nan Duan3† 1Harbin Institute of Technology, Harbin, China, 2Northeastern University, Shenyang, China 3Microsoft Research Asia, 4Intel Labs, Cognitive Computing Research {xxu,car}@ir.hit.edu.cn, libei_neu@outlook.com {chenfei.wu,nanduan}@microsoft.com, shao-yen.tseng@intel.com {anahita.bhiwandiwalla,shachar.rosenman,vasudev.lal}@intel.com Abstract Two-Tower Vision-Language (VL) models have shown promising improvements on vari- ous downstream VL tasks. Although the most advanced work improves performance by build- ing bridges between encoders, it suffers from in- effective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit dif- ferent levels of uni-modal semantic knowledge. In this work, we propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels. The managers intro- duced in each cross-modal layer can adaptively aggregate uni-modal semantic knowledge to fa- cilitate more comprehensive cross-modal align- ment and fusion. ManagerTower outperforms previous strong baselines both with and with- out Vision-Language Pre-training (VLP). With only 4M VLP data, ManagerTower achieves su- perior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. Code and checkpoints are available at https://github.com/ LooperXX/ManagerTower. 1 Introduction In recent years, there has been a growing interest in the field of Vision-Language (VL) representa- tion learning due to the development of Vision- Language Pre-training (VLP) techniques. VLP aims to learn transferable multi-modal knowledge from large-scale image-text pairs, which can fur- ther improve the performance of various down- stream VL tasks, such as visual question answer- ing (Goyal et al., 2017), visual entailment (Xie et al., 2019), visual reasoning (Suhr et al., 2019), and image-text retrieval (Young et al., 2014). Visual and textual modalities in VL models are typically processed by uni-modal encoders and sub- sequently fused in a cross-modal encoder. This ∗Contribution during internship at Microsoft. †Contact Person (a) BridgeTower (b) ManagerTower Textual Encoder Cross-Modal Encoder Visual Encoder Figure 1: Brief illustrations of BridgeTower and Man- agerTower. Hollow arrows indicate the transmission of multi-layer uni-modal representations in ManagerTower instead of layer-by-layer transmission in BridgeTower. general architecture can be referred to as the Two- Tower architecture. METER (Dou et al., 2022) and BridgeTower (Xu et al., 2022) are two representa- tive Two-Tower VL models. METER uses CLIP- ViT (Radford et al., 2021) and RoBERTa (Liu et al., 2019b) as pre-trained uni-modal encoders, but it ig- nores different levels of uni-modal semantic knowl- edge in them and only feeds the last-layer outputs of each uni-modal encoder into the cross-modal encoder. In an effort to address this issue, as illus- trated in Figure 1(a), BridgeTower connects multi- ple top uni-modal layers with each cross-modal layer in a layer-by-layer fashion to exploit uni- modal semantic knowledge at different levels. In this work, we build upon the research of BridgeTower and advance it in two …

Content truncated to 500 words