Character-centric Story Visualization via Visual Planning and Token Alignment

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8259 - 8272 December 7-11, 2022 ©2022 Association for Computational Linguistics Character-Centric Story Visualization via Visual Planning and Token Alignment Hong Chen1∗, Rujun Han3, Te-Lin Wu2, Hideki Nakayama1 and Nanyun Peng2 The University of Tokyo1, University of California, Los Angeles2, AWS AI Labs3 {chen, nakayama}@nlab.ci.i.u-tokyo.ac.jp rujunh@amazon.com, {telinwu,violetpeng}@cs.ucla.edu Abstract Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story. This task requires machines to 1) understand long text inputs and 2) produce a globally con- sistent image sequence that illustrates the con- tents of the story. A key challenge of con- sistent story visualization is to preserve char- acters that are essential in stories. To tackle the challenge, we propose to adapt a recent work that augments Vector-Quantized Varia- tional Autoencoders (VQ-VAE) with a text-to- visual-token (transformer) architecture. Specif- ically, we modify the text-to-visual-token mod- ule with a two-stage framework: 1) character token planning model that predicts the visual tokens for characters only; 2) visual token completion model that generates the remain- ing visual token sequence, which is sent to VQ-VAE for finalizing image generations. To encourage characters to appear in the images, we further train the two-stage framework with a character-token alignment objective. Exten- sive experiments and evaluations demonstrate that the proposed method excels at preserv- ing characters and can produce higher qual- ity image sequences compared with the strong baselines. Code can be found in https: //github.com/PlusLabNLP/VP-CSV 1 Introduction Text-to-image generation (Ramesh et al., 2021; Ding et al., 2021; Qiao et al., 2019), as a benchmark task to test AI systems’ multi-modal capability, has been widely studied over the past decade. The task takes a short sentence or phrase as input and out- puts a single image illustrating the content of the input text. One of the latest successful works is DALL-E (Ramesh et al., 2021), which efficiently encodes and decodes images with discrete visual tokens and achieves remarkable text-to-image gen- ∗Work done when the author was visiting UCLA. Figure 1: Example of generated images from previ- ous models (CP-CSV and VLC), the VQ-VAE with transformers (VQ-VAE-LM) baseline, and our proposed model (VP-CSV). We highlight the characters in each sentence in red. We can see that the generated image sequence by VP-CSV achieves the best image quality and character preservation. eration performance by prefixing visual tokens with natural language inputs. Recently, the story visualization task has at- tracted increasing research interests with a variety of potential and promising applications such as au- tomatic production of movie clips and animations from written scripts. Story visualization is more challenging than the original text-to-image genera- tion as it requires models to generate a sequence of images that visually illustrate a story composed of at least several sentences. Machines need to under- stand long story texts and simultaneously generate images with consistent scenes and characters de- scribed in the story coherently. Character is one of the essential elements of a story since they …

Content truncated to 500 words