Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6772–6786 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6772 Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization Adyasha Maharana Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill {adyasha, mbansal}@cs.unc.edu Abstract While much research has been done in text- to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image se- quence (or visual story). Prior work in this domain has shown that there is ample room for improvement in the generated image se- quence in terms of visual quality, consistency and relevance. In this paper, we first ex- plore the use of constituency parse trees us- ing a Transformer-based recurrent architecture for encoding structured input. Second, we augment the structured input with common- sense information and study the impact of this external knowledge on the generation of vi- sual story. Third, we also incorporate visual structure via bounding boxes and dense cap- tioning to provide feedback about the char- acters/objects in generated images within a dual learning setup. We show that off-the- shelf dense-captioning models trained on Vi- sual Genome can improve the spatial structure of images from a different target domain with- out needing fine-tuning. We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in visual qual- ity. Finally, we provide an analysis of the lin- guistic and visuo-spatial information.1 1 Introduction Story Visualization is an emerging area of research with several potentially interesting applications such as visualization of educational materials, as- sisting artists with web-comic creation etc. Each story consists of a sequence of images along with a sequence of captions describing the content of the 1Code and data: https://github.com/ adymaharana/VLCStoryGan. Ground Truth DuCo Captions VLC Petty asks whether it is because of cookies. Eddy denies with his hands. Petty hands her cookies to Eddy. Petty gives her cookies to Loopy and Crong. Crong sighs. Figure 1: Example of Generated Images from our model VLC-STORYGAN and Duco-StoryGAN (Ma- harana et al., 2021) for the PororoSV dataset. images. The goal of the task is to reproduce the images given the captions. It is more challenging than conventional text-to-image generation (Reed et al., 2016) because the generative model needs to identify the narrative structure expressed in the sequence of captions and translate it into a story of images. Some critical features of a good story in- clude consistent character and background appear- ances, relevance to individual captions as well as overall story, and coherent narrative. While recent text-to-image models (Ramesh et al., 2021; Cho et al., 2020; Li et al., 2019a) are successfully gen- erating high-quality images, they are not directly designed for narrative understanding over sequen- tial text. Hence, story visualization necessitates independent research towards developing genera- tive …

Content truncated to 500 words