PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12890–12903 July 9-14, 2023 ©2023 Association for Computational Linguistics PuMer: Pruning and Merging Tokens for Efficient Vision Language Models Qingqing Cao Bhargavi Paranjape {qicao,bparan,hannaneh}@cs.washington.edu University of Washington Hannaneh Hajishirzi Abstract Large-scale vision language (VL) models use Transformers to perform cross-modal interac- tions between the input text and image. These cross-modal interactions are computationally expensive and memory-intensive due to the quadratic complexity of processing the input image and text. We present PuMer1: a token reduction framework that uses text-informed Pruning and modality-aware Merging strate- gies to progressively reduce the tokens of in- put image and text, improving model inference speed and reducing memory footprint. PuMer learns to keep salient image tokens related to the input text and merges similar textual and visual tokens by adding lightweight token re- ducer modules at several cross-modal layers in the VL model. Training PuMer is mostly the same as finetuning the original VL model but faster. Our evaluation for two vision language models on four downstream VL tasks shows PuMer increases inference throughput by up to 2x and reduces memory footprint by over 50% while incurring less than a 1% accuracy drop. 2 1 Introduction Large-scale vision language models (Dou et al., 2022; Wang et al., 2022; Zeng et al., 2021; Kim et al., 2021; Wang et al., 2021; Zhang et al., 2021) have shown substantial progress on many vision language tasks such as visual question answer- ing, natural language visual reasoning, and vi- sual entailment. However, state-of-the-art lan- guage and vision models are memory intensive and computationally expensive because they use multi-layer self-attention between many language and vision input tokens (small image patches) with quadratic complexity. This inefficiency limits high- throughput cloud deployments and makes it infea- sible to run on resource-constrained devices. 1Pronounced as “puma” 2Code is available at https://github.com/ csarron/PuMer. 2.#What#sport#are#they#playing? Prune# Merge 1.#How#many#people#are#playing? Prune# Merge Soccer Four Figure 1: PuMer applies token reduction to VL models via pruning and merging. PuMer makes VL models run faster by text-informed image pruning to remove text- irrelevant image tokens and modality-aware merging to compress similar input tokens. The key source of inefficiency in deep VL mod- els is that these models need to process the entire input image and text tokens over all the layers. Our intuition is that the input image contains redundant information and only parts of the image (salient regions, referred by the text) are required and re- lated to the end task. For example, in Figure 1, most of the image content (the four persons, field) is not needed except for the bottom-center soccer region to answer the visual question “What sport are they playing?”. This paper advocates using the correlations between image and text modalities to reduce tokens for VL problems. In the vision-only or text-only domains, re- searchers have shown that reducing image or text tokens can improve the model computational com- plexity through pruning (Liang et al., 2021; Rao et …

Content truncated to 500 words