Diverse and Relevant Visual Storytelling with Scene Graph Embeddings

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 420–430 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 420 Diverse and Relevant Visual Storytelling with Scene Graph Embeddings Xudong Hong12, Rakshith Shetty1, Asad Sayeed3, Khushboo Mehra2, Vera Demberg2 and Bernt Schiele1 1Dept. of Computer Vision and Machine Learning, MPI Informatics 2Dept. of Language Science and Technology, Saarland University 3Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg {xhong,kmehra,vera}@coli.uni-saarland.de {rshetty,schiele}@mpg.mpi-inf.de, asad.sayeed@gu.se Abstract A problem in automatically generated stories for image sequences is that they use overly generic vocabulary and phrase structure and fail to match the distributional characteristics of human-generated text. We address this problem by introducing explicit representa- tions for objects and their relations by extract- ing scene graphs from the images. Utiliz- ing an embedding of this scene graph enables our model to more explicitly reason over ob- jects and their relations during story genera- tion, compared to the global features from an object classifier used in previous work. We apply metrics that account for the diversity of words and phrases of generated stories as well as for reference to narratively-salient image features and show that our approach outper- forms previous systems. Our experiments also indicate that our models obtain competitive re- sults on reference-based metrics. 1 Introduction Visual storytelling is the generation of a coherent narrative from a series of images (Huang et al., 2016). In this paper, we address a particular chal- lenge in visual storytelling: reflecting human pref- erences in narrative structure, especially the choice of content words and phrases that comprise a read- able story. Humans prefer to use diverse words and phrases to construct the storyline to avoid repeti- tions within or across sentences. For example, in the human-written story in Fig. 1, very few con- tent words are repeated. However, Modi and Parde (2019) have found that recent work often gener- ate repetitive words and phrases which leads to repetitions across sentences and makes stories less diverse. For example, in the first story of Fig. 1, the model generates a verb phrase had a great time and then repeats it in the fifth sentence. These words 1Typo generated by human: ”have” instead of ”gave”. Image sequence Scene graph Story Human: Governor Brandon had an event scheduled. When he arrived at his event he was escorted in by military men. He went to the serving line and ordered some food. He even gave the server a tip. The Governor have a speech about healthcare. After speech he stayed and talked to local people about their concerns. As few of the residents presented him with a jacket to show their appreciation. Baseline: everyone was excited for the ceremony to begin . everyone was excited to be there . he was very happy to be there . we had a great time . they all had a great time . Ours: i went to the party last week . the chef was preparing the food . he was very happy …

Content truncated to 500 words