Weakly Supervised Vision-and-Language Pre-training with Relative Representations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8341–8355 July 9-14, 2023 ©2023 Association for Computational Linguistics Weakly Supervised Vision-and-Language Pre-training with Relative Representations Chi Chen1,3, Peng Li∗ 2,4, Maosong Sun∗ 1,3, Yang Liu1,2,3,4 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Shanghai Artificial Intelligence Laboratory, Shanghai, China Abstract Weakly supervised vision-and-language pre- training (WVLP), which learns cross-modal representations with limited cross-modal super- vision, has been shown to effectively reduce the data cost of pre-training while maintain- ing decent performance on downstream tasks. However, current WVLP methods use only lo- cal descriptions of images, i.e., object tags, as cross-modal anchors to construct weakly- aligned image-text pairs for pre-training. This affects the data quality and thus the effective- ness of pre-training. In this paper, we pro- pose to directly take a small number of aligned image-text pairs as anchors, and represent each unaligned image and text by its similarities to these anchors, i.e., relative representations. We build a WVLP framework based on the rela- tive representations, namely RELIT1, which collects high-quality weakly-aligned image- text pairs from large-scale image-only and text-only data for pre-training through relative representation-based retrieval and generation. Experiments on four downstream tasks show that RELIT achieves new state-of-the-art re- sults under the weakly supervised setting2. 1 Introduction Vision-and-language pre-training (VLP) (Chen et al., 2020; Zhang et al., 2021; Kim et al., 2021; Radford et al., 2021; Wang et al., 2022a) has received increasing attention in recent years for its great success on various vision-and-language tasks, such as visual question answering (Antol et al., 2015), cross-modal retrieval (Plummer et al., 2015), and image captioning (Lin et al., 2014). Different from other foundation models (Bom- masani et al., 2021) such as BERT (Devlin et al., ∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Maosong Sun (sms@tsinghua.edu.cn) 1Relative rEpresentation-based Language-Image pre- Training 2Code is available at https://github.com/THUNLP-MT/ RELIT. 2018) and MAE (He et al., 2022) that only require single-modality data, VLP models rely on large- scale aligned image-text datasets (Lin et al., 2014; Sharma et al., 2018; Ordonez et al., 2011; Krishna et al., 2017) to bridge the gap between the two modalities, which requires either extensive manual annotations or heavy data cleaning processes (Lin et al., 2014; Sharma et al., 2018). The natural dif- ficulty of obtaining paired data hinders the scale of cross-modal datasets, while the success of uni- modal pre-trained models implies the potential to exploit the unlabeled data for pre-training. There- fore, besides collecting more paired data, it is a worthwhile direction to explore how to utilize low- cost unimodal data with limited cross-modal super- vision, i.e., weakly supervised vision-and-language pre-training (WVLP). The core challenge of WVLP is to establish the connection between the two modalities without us- ing a large number of aligned image-text pairs. Ex- isting works on WVLP (Li et al., 2021b; Zhou …

Content truncated to 500 words