UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2592–2607 August 1–6, 2021. ©2021 Association for Computational Linguistics 2592 UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning Wei Li∗, Can Gao∗, Guocheng Niu∗, Xinyan Xiao∗, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang Baidu Inc., Beijing, China {liwei85,gaocan01,niuguocheng,xiaoxinyan, liuhao24,liujiachen,wu hua,wanghaifeng}@baidu.com Abstract Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other. They can only utilize single-modal data (i.e., text or image) or limited multi-modal data (i.e., image-text pairs). In this work, we pro- pose a UNIfied-MOdal pre-training architec- ture, namely UNIMO, which can effectively adapt to both single-modal and multi-modal understanding and generation tasks. Large scale of free text corpus and image collec- tions are utilized to improve the capability of visual and textual understanding, and cross- modal contrastive learning (CMCL) is lever- aged to align the textual and visual informa- tion into a unified semantic space, over a corpus of image-text pairs augmented with related images and texts. With the help of rich non-paired single-modal data, our model is able to learn more generalizable representations, by allowing textual knowl- edge and visual knowledge to enhance each other in the unified semantic space. The ex- perimental results show that UNIMO greatly improves the performance of several single- modal and multi-modal downstream tasks. Our code and pre-trained models are public at https://github.com/PaddlePaddle/ Research/tree/master/NLP/UNIMO. 1 Introduction Large-scale pre-training has drawn much atten- tion in both the community of Compute Vision (CV) and Natural Language Processing (NLP) due to its strong capability of generalization and effi- cient usage of large-scale data. Firstly in CV, a series of models were designed and pre-trained on the large-scale dataset ImageNet, such as AlexNet (Krizhevsky et al., 2017), VGG (Simonyan and ∗These authors contribute equally to this study and are listed with random order. Who is standing behind the baseball player? (a) Cocaher (b) Umpire (c) Spectator Any baseball game involves one or more umpires, who make rulings on the outcome of each play. At a minimum, one umpire will stand behind the catcher, to have a good view of the strike zone, and call balls and strikes. Additional umpires may be stationed near the other bases … from wikipedia Figure 1: An illustrative example for the necessity of unified-modal learning. We can only determine the cor- rect answer to the visual question based on the textual background information. Zisserman, 2014) and ResNet (He et al., 2016), which effectively improved the capability of im- age recognition for numerous tasks. Recent years have witnessed the burst of pre-training in NLP, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and UniLM (Dong et al., 2019), which greatly improve the capa- bilities of language understanding and generation. However, the above researches focus on the single- modal learning and can only be effectively used in single-modal (i.e., only …

Content truncated to 500 words