TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4084 - 4096 December 7-11, 2022 ©2022 Association for Computational Linguistics TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection Chaoya Jiang1, Haiyang Xu 2, Chenliang Li 2, Ming Yan 2 Wei Ye 1∗, Shikun Zhang 1∗, Bin Bi 2, Songfang Huang 2 1 National Engineering Research Center for Software Engineering, Peking University 2 DAMO Academy, Alibaba Group {jiangchaoya,wye, zhangsk}@pku.edu.cn, {shuofeng.xhy,ym119608,lcl193798,b.bi, songfang.hsf}@alibaba-inc.com Abstract Vision Transformers (ViTs) have been widely used in large-scale Vision and Language Pre- training (VLP) models. Though previous VLP works have proved the effectiveness of ViTs, they still suffer from computational efficiency brought by the long visual sequence. To tackle this problem, in this paper, we propose an effi- cient vision-and-language pre-training model with Text-Relevant Image Patch Selection, namely TRIPS, which reduces the visual se- quence progressively with a text-guided patch- selection layer in the visual backbone for ef- ficient training and inference. The patch- selection layer can dynamically compute text- dependent visual attention to identify the at- tentive image tokens with text guidance and fuse inattentive ones in an end-to-end manner. Meanwhile, TRIPS does not introduce extra parameters to ViTs. Experimental results on a variety of popular benchmark datasets demon- strate that TRIPS gain a speedup of 40% over previous similar VLP models, yet with compet- itive or better downstream task performance. 1 Introduction In recent years, Vision-Language Pre-training (VLP) (Tan and Bansal, 2019; Chen et al., 2019; Lu et al., 2019; Huang et al., 2020; Su et al., 2020; Li et al., 2020; Chen et al., 2020; Zhou et al., 2020; Li et al., 2021b; Yu et al., 2021; Li et al., 2022) has developed at an astonishing rate and become a prevalent paradigm to tackle VL tasks. Traditional VLP models (Tan and Bansal, 2019; Chen et al., 2019; Lu et al., 2019; Li et al., 2020) utilize pre- trained object detectors (Ren et al., 2015; Redmon et al., 2016; He et al., 2017) to extract region-based image features but suffer from extensive annota- tion and expensive computation of object detector training. Inspired by the success of the Vision Transformer (ViT) (Dosovitskiy et al., 2021) and its variants (Liu et al., 2021; Wu et al., 2021; Wang ∗corresponding author. et al., 2021a) in computer vision field, more recent VLP models (Li et al., 2021b; Radford et al., 2021; Kim et al., 2021; Wang et al., 2021b; Singh et al., 2021) have adopted ViT as the visual encoder or cross-modal fusion encoder without using region features from the pre-trained object detector. However, these ViT-based VLP methods are re- quired to model long visual sequences from high- resolution images for good vision understanding, with quadratic computational complexity to the length of the visual sequence. Moreover, recent efforts (Dosovitskiy et al., 2021; Touvron et al., 2021) have also begun to explore vision-language foundation models, which scale up the model and data size. This raises the necessity to decrease the high computational cost of ViT-based VLP …

Content truncated to 500 words