Style-Aware Contrastive Learning for Multi-Style Image Captioning

Findings of the Association for Computational Linguistics: EACL 2023, pages 2257–2267 May 2-6, 2023 ©2023 Association for Computational Linguistics Style-Aware Contrastive Learning for Multi-Style Image Captioning Yucheng Zhou, Guodong Long∗ Australian AI Institute, School of Computer Science, FEIT, University of Technology Sydney yucheng.zhou-1@student.uts.edu.au, guodong.long@uts.edu.au Abstract Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired lin- guistic style. However, existing methods over- look the relationship between linguistic style and visual content. To overcome this draw- back, we propose style-aware contrastive learn- ing for multi-style image captioning. First, we present a style-aware visual encoder with con- trastive learning to mine potential visual con- tent relevant to style. Moreover, we propose a style-aware triplet contrast objective to dis- tinguish whether the image, style and caption matched. To provide positive and negative sam- ples for contrastive learning, we present three retrieval schemes: object-based retrieval, RoI- based retrieval and triplet-based retrieval, and design a dynamic trade-off function to calculate retrieval scores. Experimental results demon- strate that our approach achieves state-of-the- art performance. In addition, we conduct an extensive analysis to verify the effectiveness of our method. 1 Introduction Stylized image captioning aims to generate a nat- ural language description with stylized elements for a given image (Mathews et al., 2016; Gan et al., 2017). With the advance of deep learning in human- computer interaction equipment, it has been inte- grated into many real-world applications like edu- cation robots (Zhou, 2022), visual dialog (Das et al., 2017), and vision-language navigation (Wang et al., 2019). Therefore, it has attracted more attention from academia and industry and has become one of the essential areas in the natural language process- ing (NLP) and computer vision (CV) community. However, many methods propose translating im- ages into captions of a single caption style (Math- ews et al., 2016; Gan et al., 2017). These methods need to train multiple models to handle multiple ∗Corresponding author. Style: Contradictory Caption: Love what you did with the place! Style: Happy Caption: Love what you did with the place! Style: Happy Caption: the building is so beautiful and I love it. (a) (b) (c) Figure 1: (a) the caption is generated by stylized cap- tioning model (Shuster et al., 2019); (b) and (c) show the same caption can correspond to different styles. styles, which is very inefficient. Therefore, a rising demand for stylized image captioning is to learn an efficient model to handle multiple styles simul- taneously. Recently, many efforts have been made in multi-style image captioning (Guo et al., 2019; Shuster et al., 2019; Li and Harrison, 2021). Despite their success, existing methods suffer from a drawback: They focus on accurate visual content and desired linguistic style but overlook the relation between linguistic style and visual content. As shown in Figure 1(a), the generated caption con- tains accurate visual content and the desired style, but with misinformation that is easily detectable by humans, i.e., model generates “beautiful” in the caption to cater to the style of …

Content truncated to 500 words