Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences

Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences Xudong Hong1,2,4, Asad Sayeed3, Khushboo Mehra2,4, Vera Demberg2,4 and Bernt Schiele1,4 1Dept. of Computer Vision and Machine Learning, MPI Informatics, Germany 2Dept. of Language Science and Technology and Dept. of Computer Science, Saarland University, Germany 3Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg, Sweden 4Saarland Informatics Campus, Saarbr¨ucken, Germany {xhong,kmehra,vera}@lst.uni-saarland.de schiele@mpi-inf.mpg.de, asad.sayeed@gu.se Abstract Current work on image-based story generation suffers from the fact that the existing image se- quence collections do not have coherent plots behind them. We improve visual story gen- eration by producing a new image-grounded dataset, Visual Writing Prompts (VWP). VWP contains almost 2K selected sequences of movie shots, each including 5-10 images. The image sequences are aligned with a total of 12K stories which were collected via crowd- sourcing given the image sequences and a set of grounded characters from the corresponding image sequence. Our new image sequence col- lection and filtering process has allowed us to obtain stories that are more coherent, diverse, and visually grounded compared to previous work. We also propose a character-based story generation model driven by coherence as a strong baseline. Evaluations show that our generated stories are more coherent, visually grounded, and diverse than stories generated with the current state-of-the-art model. Our code, image features, annotations and collected stories are available at https://vwprompt .github.io/. 1 Introduction In this work, we improve the quality of text stories generated by neural models from image sequences. We do so by improving the curation of the image sequences that form the basis for collecting the story/image pairs used to train the models: We build a dataset in which the images lend themselves better to telling a story. To show the usefulness of our dataset, we train a coherence- driven model where we design a coherence compo- nent inspired by entity grid models. Experiments show that our model produces more coherent, vi- sually grounded and diverse stories than previous models. Stories are essential in natural language un- derstanding and generation because they are the key mechanism for humans to understand the world (Piper et al., 2021). Automatically gener- ating good stories is a challenging task requiring various capabilities in language processing (Peng et al., 2018), event understanding (Martin et al., 2018; Hong et al., 2020), and world knowledge (Guan et al., 2020; Hsu et al., 2020) to come to- gether. Previous approaches to story generation have used different kinds of input to guide the story: Some use a textual prompt to start the story (Fan et al., 2018), yet others involve describing a sequence of images to direct the story (Huang et al., 2016). We choose to work inside the latter family of approaches in order to exploit the rich information contained in image sequences and to prevent suffering from the symbol grounding problem (Harnad, 1990). Research on visual narratives shows how it would be possible to construct the sort of dataset we propose: Image sequences should consist of a series of …

Content truncated to 500 words