End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10799 - 10810 December 7-11, 2022 ©2022 Association for Computational Linguistics End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching Chi Chen1,2,5, Peng Li∗ 3, Maosong Sun1,2,5, Yang Liu∗ 1,2,3,4,5,6 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Beijing National Research Center for Information Science and Technology 3Institute for AI Industry Research, Tsinghua University, Beijing, China 4Beijing Academy of Artificial Intelligence, Beijing, China 5International Innovation Center of Tsinghua University, Shanghai, China 6Quan Cheng Laboratory Abstract Recently there has been an emerging interest in unsupervised vision-and-language pre-training (VLP) that learns multimodal representations without parallel image-caption data. These pio- neering works significantly reduce the cost of VLP on data collection and achieve promising results compared to supervised VLP. However, existing unsupervised VLP methods take as in- put pre-extracted region-based visual features from external object detectors, which both lim- its flexibility and reduces computational effi- ciency. In this paper, we explore end-to-end unsupervised VLP with a vision encoder to di- rectly encode images. The vision encoder is pre-trained on image-only data and jointly opti- mized during multimodal pre-training. To fur- ther enhance the learned cross-modal features, we propose a novel pre-training task that pre- dicts which patches contain an object referred to in natural language from the encoded visual features. Extensive experiments on four vision- and-language tasks show that our approach out- performs previous unsupervised VLP methods and obtains new state-of-the-art results1. 1 Introduction Vision-and-language pre-training (VLP) (Lu et al., 2019; Li et al., 2019; Chen et al., 2020; Kim et al., 2021; Li et al., 2021c; Zhang et al., 2021; Radford et al., 2021; Ramesh et al., 2021) has achieved great success on a wide range of vision-and-language tasks, e.g., visual question answering (Zhang et al., 2021), image-text retrieval (Radford et al., 2021) and text-to-image generation (Ramesh et al., 2021). The major challenge for VLP is how to bridge the gap between the representations of vision and language modalities, which is typically ad- ∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn) 1Code is available at https://github.com/THUNLP-MT/ E2E-UVLP dressed by training on large-scale parallel image- text datasets (Lin et al., 2014; Krishna et al., 2017; Sharma et al., 2018; Ordonez et al., 2011) with spe- cially designed pre-training tasks. However, these datasets require either extensive human annotations or massive data cleaning efforts, making them dif- ficult to collect, especially when compared to the large amount of unimodal data. To alleviate this problem, there has recently emerged some works exploring unsupervised vision-and-language pre-training (UVLP), where only non-parallel image and text data is lever- aged (Li et al., 2021b; Zhou et al., 2022). Specif- ically, Li et al. (2021b) propose to use image re- gion features and their detected object tags pro- duced by an object detector as pesudo-parallel pairs to bridge the gap between the two modali- ties. Zhou et al. (2022) further enrich the training data with retrieved text pieces based on object tags …

Content truncated to 500 words