Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2383 - 2395 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation Wenliang Dai1, Lu Hou2, Lifeng Shang2, Xin Jiang2, Qun Liu2, Pascale Fung1 1Hong Kong University of Science and Technology, 2Huawei Noah’s Ark Lab wdaiai@connect.ust.hk, pascale@ece.ust.hk, {houlu3, shang.lifeng, jiang.xin, qun.liu}@huawei.com Abstract The recent large-scale vision-language pre- training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. De- spite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this prob- lem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental re- sults show that the resulting model has strong zero-shot performance on multimodal genera- tion tasks, such as open-ended visual question answering and image captioning. For exam- ple, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with 7× fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks. 1 Introduction Recent large-scale dual-stream Vision-Language Pre-training (VLP) models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), have shown remarkable performance on various down- stream multimodal alignment tasks, e.g., image- text retrieval and image classification. These mod- els are pre-trained using cross-modal contrastive learning on tremendous image-text pairs and learn strong multimodal representations. Despite their success, as mentioned by Radford et al. (2021), their text encoder is relatively weak by only having a discriminative multimodal pre-training objective, 𝑉𝐿𝑃𝐼𝑚𝑔 𝑉𝐿𝑃𝑇𝑒𝑥𝑡 𝑁𝐿𝑃𝐷���𝑚���𝑐 𝑁𝐿𝑃𝐷𝑒𝑐 Knowledge Distillation Training Phase (VLKD) Zero-Shot Inference 𝑉𝐿𝑃𝐼𝑚𝑔 What small cloth is used to wipe the face or eating this meal? Answer: [mask]. ... ? Answer: A napkin. 𝑁𝐿𝑃𝐷���𝑚���𝑐 𝑁𝐿𝑃𝐷𝑒𝑐 𝑉𝐿𝑃𝐼𝑚𝑔 A picture of [mask]. A picture of a bouquet of white flowers in a glass vase. 𝑁𝐿𝑃𝐷���𝑚���𝑐 𝑁𝐿𝑃𝐷𝑒𝑐 × 𝒎 Visual Question Answering Image Captioning × 𝒎��� Figure 1: Intuition of our proposed approach. After VLKD, the model can fill in the masked locations with meaningful words to describe the image without further finetuning. Moreover, it can answer questions with proper reasoning over the given images and pre-trained knowledge inside PLMs, e.g., a napkin is for wiping the face at meals. which makes them incompetent on generative mul- timodal tasks such as image captioning and open- ended visual question answering (VQA). Meanwhile, the Transformer-based (Vaswani et al., 2017) auto-regressive large-scale pre-trained language models (PLMs), such as GPT (Radford and Narasimhan, 2018; Brown et al., 2020), have been dominating in the natural language genera- tion (NLG) tasks. These models are usually trained with causal self-attention, which only allows the model to attend to past outputs (unidirectional) to satisfy …

Content truncated to 500 words