Plot and Rework: Modeling Storylines for Visual Storytelling

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4443–4453 August 1–6, 2021. ©2021 Association for Computational Linguistics 4443 Plot and Rework: Modeling Storylines for Visual Storytelling Chi-Yang Hsu1,3∗, Yun-Wei Chu2∗, Ting-Hao (Kenneth) Huang1, Lun-Wei Ku3 Pennsylvania State University 1, Purdue University 2, Institute of Information Science, Academia Sinica3 {cxh5438, txh710}@psu.edu {chu198}@purdue.edu {lwku}@iis.sinica.edu.tw Abstract Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Auto- mated visual storytelling (VIST) models, how- ever, make poor use of external knowledge and iterative generation when attempting to create stories. This paper introduces PR-VIST, a framework that represents the input image se- quence as a story graph in which it finds the best path to form a storyline. PR-VIST then takes this path and learns to generate the final story via a re-evaluating training process. This framework produces stories that are superior in terms of diversity, coherence, and humanness, per both automatic and human evaluations. An ablation study shows that both plotting and re- working contribute to the model’s superiority. 1 Introduction Writing a story is a complicated task. Human writ- ers use their knowledge to tie all the disjointed ele- ments, such as people, items, actions, or locations, together to form a coherent storyline. Writers also re-evaluate their work constantly during the writ- ing process, and sometimes even alter their writing goals in the middle of a draft. Flower and Hayes (1981) characterize a solo writer’s cognitive pro- cess as a series of components in which the writer’s own knowledge is described as the long-term mem- ory, and the planning, translating, and reviewing steps can occur in a recursive, interconnected man- ner. These creative steps are essential to human writing. However, automated visual storytelling (VIST) models that compose stories given five im- ages (Huang et al., 2016) do not make extensive use of human knowledge to tie the elements to- ∗* denotes equal contribution gether, nor do they use human insight to evaluate the outputs and guide the generation process. As for linking elements, most works generate vi- sual stories in an end-to-end fashion (Huang et al., 2016; Kim et al., 2018), treating the task as a straightforward extension of image captioning. Re- cent works have begun to use relations between entities to improve visual storytelling, but often narrow in a particular subset of relations, such as relations between elements within the same im- age (Yang et al., 2019), relations between two ad- jacent images (Hsu et al., 2020), or relations be- tween scenes (Wang et al., 2020). The full poten- tial of rich real-world knowledge and intra-image relations have yet to be fully utilized. As for re- evaluation, recent work uses reward systems (Wang et al., 2018b; Hu et al., 2019) or estimated topic co- herence (Wang et al., 2019) to automatically assess the output story and guide the generation process. However, these approaches are often optimized to- …

Content truncated to 500 words