Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication

Proceedings of the 28th International Conference on Computational Linguistics, pages 2250–2260 Barcelona, Spain (Online), December 8-13, 2020 2250 Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication Ruize Wang1, Zhongyu Wei2,3∗, Ying Cheng1, Piji Li4, Haijun Shan5, Ji Zhang5, Qi Zhang6, Xuanjing Huang6 1 Academy for Engineering and Technology, Fudan University, China 2School of Data Science, Fudan University, China 3Research Institute of Intelligent and Complex Systems, Fudan University, China 4Tencent AI Lab, China, 5Zhejiang Lab, China 6School of Computer Science, Fudan University, China {rzwang18,zywei,chengy18,qz,xjhuang}@fudan.edu.cn; lipiji.pz@gmail.com; shanhaijun@zhejianglab.com; Ji.Zhang@usq.edu.au Abstract Visual storytelling aims to generate a narrative paragraph from a sequence of images automati- cally. Existing approaches construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoher- ent content. In this paper, we propose a new way for visual storytelling by introducing a topic description task to detect the global semantic context of an image stream. A story is then con- structed with the guidance of the topic description. In order to combine the two generation tasks, we propose a multi-agent communication framework that regards the topic description genera- tor and the story generator as two agents and learn them simultaneously via iterative updating mechanism. We validate our approach on VIST dataset, where quantitative results, ablations, and human evaluation demonstrate our method’s good ability in generating stories with higher quality compared to state-of-the-art methods. 1 Introduction Image-to-text generation is an important topic in artificial intelligence (AI) which connects computer vision (CV) and natural language processing (NLP). Popular tasks include image captioning (Karpathy and Fei-Fei, 2015; Ren et al., 2017; Vinyals et al., 2017) and question answering (Antol et al., 2015; Yu et al., 2017b; Singh et al., 2019), aiming at generating a short sentence or a phrase conditioned on certain visual information. With the development of deep learning and reinforcement learning models, recent years witness promising improvement of these tasks for single-image-to-single-sentence generation. Visual storytelling moves one step further, extending the input and output dimension to a sequence of images and a sequence of sentences. It requires the model to understand the main idea of an image stream and generate coherent sentences. Most of existing methods (Huang et al., 2016; Liu et al., 2017; Yu et al., 2017a; Wang et al., 2018a; Wang et al., 2020) for visual storytelling extend approaches of image captioning without considering topic information of the image sequence, which causes the problem of generating semantically incoherent content. An example of visual storytelling can be seen in Figure 1. An image stream with five images about a car accident is presented accompanied with two stories. One is constructed by a human annotator and the other is produced by an automatic storytelling approach. There are two problems with the machine generated story. First, the sentiment expressed in the text is inappropriate. In face of a terrible car accident, the model uses some words with positive emotion, i.e., “excited” and “fun”. Second, some sentence …

Content truncated to 500 words