mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7241 - 7259 December 7-11, 2022 ©2022 Association for Computational Linguistics mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections Chenliang Li∗, Haiyang Xu∗, Junfeng Tian, Wei Wang, Ming Yan†, Bin Bi†, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si DAMO Academy, Alibaba Group {lcl193798, shuofeng.xhy, tjf141457, hebian.ww, ym119608, b.bi, yejiabo.yjb, hehong.chh, guohai.xgh, zhengzhi.cz, zj122146, songfang.hsf, f.huang, jingren.zhou, luo.si}@alibaba-inc.com Abstract Large-scale pre-trained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of down- stream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and genera- tion. Most existing pre-trained models suffer from inefficiency and linguistic signal over- whelmed by long visual sequences in cross- modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross- modal skip-connections. mPLUG is pre-trained end-to-end on large- scale image-text pairs with both discrimina- tive and generative objectives. It achieves state-of-the-art results on a wide of vision-language downstream tasks,range in- cluding image captioning, image-text re- trieval, visual grounding and visual ques- tion answering. mPLUG also demonstrates strong zero-shot transferability on vision- language and video-language tasks. The code and pre-trained models are available at https://github.com/alibaba/AliceMind. 1 Introduction Large-scale pre-training of vision-language models have recently received tremendous success on a wide range of cross-modal tasks (Tan and Bansal, 2019; Chen et al., 2020; Huang et al., 2020; Li et al., 2020b; Yu et al., 2021; Li et al., 2021b; Wang et al., 2021c). Such vision-language mod- els learn cross-modal representations from a quan- tity of image-text pairs by aligning the visual and linguistic modalities. The key to learning vision- language models is finding a good alignment be- tween the two modalities to close the semantic gap in-between. ∗ Equal contribution † Corresponding authors Figure 1: Illustration of two conventional cross-modal fusion networks and our proposed cross-modal skip- connected network. We compare the running time and performance of different fusion networks, where the fusion layers, image encoder and text encoder are all kept the same. The running time is the total forward time of 100 samples in different fusion networks. Most recent work (Huang et al., 2020; Wang et al., 2021c; Li et al., 2021b; Kim et al., 2021; Dou et al., 2021) proposes a direct alignment be- tween the image and text representations in an end- to-end manner, which turns out to perform well on many vision-language tasks. These models extract finer-grained visual representation with a long se- quence of image patches or grids for good vision understanding (Dou et al., 2021). However, there are two significant problems in fusing the asymmetric information caused by long visual sequences: 1) vanishing information: the caption text in widely-used image-text pre-training data is usually short and highly abstract while more detailed and diverse information can be extracted from images as long visual sequences. …

Content truncated to 500 words