PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11104 - 11117 December 7-11, 2022 ©2022 Association for Computational Linguistics PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models Yuan Yao1∗, Qianyu Chen1∗, Ao Zhang2, Wei Ji2 Zhiyuan Liu1†, Tat-Seng Chua2, Maosong Sun1† 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology Innovation Center of Tsinghua University, Shanghai, China 2Sea-NExT Joint Lab, Singapore School of Computing, National University of Singapore, Singapore yaoyuanthu@163.com socqyc@gmail.com Abstract Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models with- out reliance on object detectors are becoming the mainstream due to their superior compu- tation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in ex- plicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehen- sion and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre- training, and also enables flexible prompt tun- ing for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehen- sion and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL. 1 Introduction Recent progress on self-supervised learning has led to powerful vision-language pre-training (VLP) models that achieve state-of-the-art performance on a wide range of cross-modal tasks (Lu et al., 2019; ∗ indicates equal contribution † Corresponding authors: Z.Liu (liuzy@tsinghua.edu.cn), M.Sun (sms@tsinghua.edu.cn) Li et al., 2020c; Radford et al., 2021; Zhang et al., 2021; Kamath et al., 2021). Typically, VLP models are first pre-trained on large-scale image-text data to learn universal cross-modal representations, and then fine-tuned to adapt to downstream tasks (Bom- masani et al., 2021). While most traditional VLP models heavily rely on external object detectors to obtain the visual inputs (Lu et al., 2019; Su et al., 2020; Li et al., 2020c; Zhang et al., 2021), recently there is a growing interest in VLP models that re- move the reliance on object detectors due to their superior computation efficiency and competitive performance (Li et al., 2021; Kim et al., 2021; Rad- ford et al., 2021; Kamath et al., 2021). However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling. The drawback hinders successful handling of vision-language (VL) tasks which are inherently object-centric, where deep understand- ing of objects and their interactions plays an essen- tial role (Antol et al., 2015; Plummer et al., 2015; Krishna et al., 2017; Hudson and Manning, 2019). Therefore, it is typically difficult for detector-free1 VLP models to handle various position-sensitive …

Content truncated to 500 words