UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

Findings of the Association for Computational Linguistics: ACL 2022, pages 3187 - 3201 May 22-27, 2022 c⃝2022 Association for Computational Linguistics UNIMO-2: End-to-End Unified Vision-Language Grounded Learning Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang Baidu Inc., Beijing, China {liwei85,gaocan01,niuguocheng,xiaoxinyan, liuhao24,liujiachen,wu_hua,wanghaifeng}@baidu.com Abstract Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified- modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic align- ment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/. 1 Introduction Large-scale pre-training has drawn much attention in the community of Computer Vision (CV), Natu- ral Language Processing (NLP) and Multi-Modal (MM) due to its strong capability of generalization and efficient usage of large-scale data. However, in the existing literature, the work on vision, lan- guage and vision-language representation learning are mostly studied separately with different train- ing data sources. In the vision domain, pre-training on large-scale image corpus such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al., 2020) and JFT-300M (Dosovitskiy et al., 2020) has proven to be critical for learning transferring vi- sual representation for various downstream tasks. In NLP, pre-training on easily-accessible unanno- tated text corpora greatly improves the capabilities of language understanding and generation (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019). Pre-training has also become the de-facto approach in vision-language modeling (Lu et al., 2019; Chen et al., 2020c; Li et al., 2020, 2019a; Yu et al., 2020). However, existing VLP methods require a mas- sive amount of aligned image-text pairs which are costly to collect and hard to scale up. The large volumes of image corpus in CV and text corpus in NLP cannot be effectively utilized. Thus, the scalability and performance upper limit of exist- ing VLP methods are largely restricted. As they only learn joint vision-language representations on image-text pairs, they are also difficult to be effec- tively adapted to visual and textual tasks (Li et al., 2021b; Lin et al., 2020). To address the limitations, we propose a new end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on various types of corpora, including …

Content truncated to 500 words