Attractive Storyteller: Stylized Visual Storytelling with Unpaired Text

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11053–11066 July 9-14, 2023 ©2023 Association for Computational Linguistics Attractive Storyteller: Stylized Visual Storytelling with Unpaired Text Dingyi Yang, Qin Jin∗ School of Information, Renmin University of China {yangdingyi, qjin}@ruc.edu.cn Abstract Most research on stylized image captioning aims to generate style-specific captions using unpaired text, and has achieved impressive per- formance for simple styles like positive and negative. However, unlike previous single- sentence captions whose style is mostly em- bodied in distinctive words or phrases, real- world styles are likely to be implied at the syn- tactic and discourse levels. In this work, we introduce a new task of Stylized Visual Story- telling (SVST), which aims to describe a photo stream with stylized stories that are more ex- pressive and attractive. We propose a multi- tasking memory-augmented framework called StyleVSG, which is jointly trained on factual vi- sual storytelling data and unpaired style corpus, achieving a trade-off between style accuracy and visual relevance. Particularly for unpaired stylized text, StyleVSG learns to reconstruct the stylistic story from roughly parallel visual inputs mined with the CLIP1 model, avoiding problems caused by random mapping in previ- ous methods. Furthermore, a memory module is designed to preserve the consistency and co- herence of generated stories. Experiments show that our method can generate attractive and co- herent stories with different styles, such as fairy tale, romance, and humor. The overall perfor- mance of our proposed StyleVSG surpasses state-of-the-art methods on both automatic and human evaluation metrics 2. 1 Introduction Over the years, Image Captioning has made re- markable progress (Xu et al., 2015; Guo et al., 2020; Hu et al., 2022a). Factual image captioning focuses on generating objective and neutral descrip- tions of image content without considering style characteristics. However, when describing images, ∗* Corresponding Author. 1https://openai.com/blog/clip/ 2The code will be released at https://github.com/ DingyiYang/StyleVSG. Factual Visual Story: Check out our new vacation house by the lake. It has a great view of the water. We can go out in our boat and travel around. Then park over by the shore to do some fishing. It is nice to have a bbq at the end of the day . Stylized Captions: … Pos:(3) Nice man and woman in a boat through water. Neg:(3) Two stupid people on a lake in a rowboat. … Factual Captions: (1) A large house sits on stilts in the water. (2) Two people are outside by a river. (3) A man and woman on a lake in a rowboat. (4) A couple that is standing near a boat. (5) The man is taking the lid of the grill. Stylized Visual Story (SVST task): Fairy Tale: Once upon a time, there was a fisherman and his wife lived in a little cottage close by river. And when they came to the lake, there was a great deal of water. Then they rowed out into the lake. They rowed on till it came to an …

Content truncated to 500 words