MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11908–11922 July 9-14, 2023 ©2023 Association for Computational Linguistics MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning Bang Yang1,2∗, Fenglin Liu3∗, Xian Wu4, Yaowei Wang2, Xu Sun5†, and Yuexian Zou1† 1ADSPLAB, School of ECE, Peking University 2Peng Cheng Laboratory 3University of Oxford 4Tencent Jarvis Lab 5School of Computer Science, Peking University {yangbang, zouyx}@pku.edu.cn; fenglin.liu@eng.ox.ac.uk Abstract Supervised visual captioning models typically require a large scale of images or videos paired with descriptions in a specific language (i.e., the vision-caption pairs) for training. However, collecting and labeling large-scale datasets is time-consuming and expensive for many sce- narios and languages. Therefore, sufficient la- beled pairs are usually not available. To deal with the label shortage problem, we present a simple yet effective zero-shot approach Mul- tiCapCLIP that can generate visual captions for different scenarios and languages without any labeled vision-caption pairs of downstream datasets. In the training stage, MultiCapCLIP only requires text data for input. Then it con- ducts two main steps: 1) retrieving concept prompts that preserve the corresponding do- main knowledge of new scenarios; 2) auto- encoding the prompts to learn writing styles to output captions in a desired language. In the testing stage, MultiCapCLIP instead takes visual data as input directly to retrieve the con- cept prompts to generate the final visual de- scriptions. The extensive experiments on image and video captioning across four benchmarks and four languages (i.e., English, Chinese, Ger- man, and French) confirm the effectiveness of our approach. Compared with state-of-the- art zero-shot and weakly-supervised methods, our method achieves 4.8% and 21.5% abso- lute improvements in terms of BLEU@4 and CIDEr metrics. Our code is available at https: //github.com/yangbang18/MultiCapCLIP. 1 Introduction Visual captioning targets to first 1) understand the information of visual inputs, which are typically videos or images, and then 2) produces a corre- sponding textual sentence describing the visual ob- jects/attributes/relationships. Visual captioning has drawn remarkable attention from natural language ∗Equal contribution. †Corresponding authors. processing and computer vision fields due to its wide applications, e.g., cross-modal retrieval (Luo et al., 2022; Cheng et al., 2023b) and helping the visually impaired (Çaylı et al., 2021). Currently, visual captioning models based on the encoder- decoder framework (Huang et al., 2020; Liu et al., 2020; Yang et al., 2021; Zhang et al., 2021; Hu et al., 2022; Lin et al., 2022) have achieved tremen- dous progress in advancing the state-of-the-art. These models are usually trained with full supervi- sion and rely on large-scale humanly-annotated training data (i.e., vision-caption pairs), which needs expensive labeling work. In particular, when it comes to Non-English caption systems, it is challenging to collect and label sufficient vision- caption pairs in a timely manner, which prevents such encoder-decoder models from rapid deploy- ment in different scenarios and languages. To deal with the shortage of labeled pairs, we propose the MultiCapCLIP - a prompt-based natu- ral language auto-encoder. As shown in Figure 1, MultiCapCLIP only requires textual input …

Content truncated to 500 words