Improving Generation and Evaluation of Visual Stories via Semantic Consistency

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2427–2442 June 6–11, 2021. ©2021 Association for Computational Linguistics 2427 Improving Generation and Evaluation of Visual Stories via Semantic Consistency Adyasha Maharana Darryl Hannan Department of Computer ScienceMohit Bansal University of North Carolina at Chapel Hill {adyasha, dhannan, mbansal}@cs.unc.edu Abstract Story visualization is an underexplored task that falls at the intersection of many impor- tant research directions in both computer vi- sion and natural language processing. In this task, given a series of natural language cap- tions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced re- current generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a num- ber of improvements to prior modeling ap- proaches, including (1) the addition of a dual learning framework that utilizes video caption- ing to reinforce the semantic alignment be- tween the story and generated images, (2) a copy-transform mechanism for sequentially- consistent story visualization, and (3) MART- based transformers to model complex interac- tions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not ac- curately reflect performance. Therefore, we also provide an exploration of evaluation met- rics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to cap- tions, and the diversity of the generated im- ages. We also present correlation experiments of our proposed automated metrics with hu- man evaluations.1 1 Introduction While generative adversarial networks (GANs) have achieved impressive results on a variety of 1Code and data: https://github.com/ adymaharana/StoryViz. The car to carry freight trains to ride Pororo and friends starts on road. The car to carry freight trains is riding across the snow-covered field. Pororo is complaining to Crong on the field. Pororo asks the car on the snow-covered field. The car told on the snow-covered field. Ground Truth Generated Captions Figure 1: Illustration of the Pororo-SV dataset (Cap- tions & Ground Truth) and the corresponding images generated from our model (Generated). image generation tasks (Zhu et al., 2019; Qiao et al., 2019), the task of story visualization (Li et al., 2019b) is a variation of image generation that is more challenging and underexplored. In this set- ting, there is a story which consists of a sequence of images along with captions describing the content of the images, e.g., a web comic. The goal of the task is to reproduce the images given the captions (Figure 1). The benefits of investigating this task are far reaching. It combines two interesting and challenging sub-areas: text-to-image synthesis and narrative understanding, …

Content truncated to 500 words