DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training

Findings of the Association for Computational Linguistics: ACL 2022, pages 2552 - 2566 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training Luyang Huang Guocheng Niu Jiachen Liu Baidu Inc., Beijing, ChinaXinyan Xiao and Hua Wu {huangluyang,niuguocheng,liujiachen,xiaoxinyan, wu_hua}@baidu.com Abstract Due to the limitations of the model structure and pre-training objectives, existing vision-and- language generation models cannot utilize pair- wise images and text through bi-directional gen- eration. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and gener- ation, we further design a novel commitment loss. We compare pre-training objectives on image captioning and text-to-image generation datasets. Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the vari- ant without the commitment loss. On the image captioning task, our model reaches better per- formance than other pre-trained systems. On text-to-image generation datasets, our model achieves better or comparable results than previ- ous state-of-the-art models. In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions. 1 Introduction Pre-trained models for vision-and-language tasks have made remarkable progress recently (Lu et al., 2019; Su et al., 2020; Chen et al., 2020). Existing pre-trained models either focus on text-to-image synthesis or image-to-text generation (Ramesh et al., 2021; Cho et al., 2021). These models are often pre-trained with image-text pairs which are aligned in semantics. However, due to the limita- tions of model structure, existing models cannot be adapted to each other. In addition, pre-training objectives are designed either for text generation conditioned on the image or image generation con- ditioned on the text, limiting the model to learn better semantic alignment from bi-directional gen- eration (Xu et al., 2021; Ding et al., 2021). Image Captioning Ground Truth: Rows of unripe bananas on a display shelf. DU-VLG: ! Several bunches of green bananas are on a shelf. w/o dual pre-training: " A bunch of green bananas sitting on a table. Ground Truth DU-VLG! w/o dual pre-training" Input: Rows of unripe bananas on a display shelf. Text-to-Image Generation Dual Tasks Figure 1: An example from COCO dataset. For image captioning, our system generates informative captions, with key words highlighted in bold. Incorrect informa- tion is underlined. For text-to-image generation, our system synthesizes vivid images aligned with captions. We argue that image-to-text and text-to-image generation appear as dual tasks, which both re- quire strong visual and textual representations aligned in the same semantic space. Images and text descriptions are of different information quan- tity and density. The images often contain more information, but are with heavy redundancy, while text descriptions are semantically condensed, but may neglect details. Uni-directional generation paradigm may induce the model to amplify this property. Take Fig.1 as an example, the uni- directional model may fail in capturing details. In- spired …

Content truncated to 500 words