Reading Between the Lines: Exploring Infilling in Visual Narratives

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1220–1229, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1220 Reading Between the Lines: Exploring Infilling in Visual Narratives Khyathi Raghavi Chandu Ruo-Ping Dong Alan W Black Language Technologies Institute, Carnegie Mellon University {kchandu, awb}@cs.cmu.edu, ruopingd@alumni.cmu.edu Abstract Generating long form narratives such as sto- ries and procedures from multiple modalities has been a long standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the sur- rounding contexts. The general seq2seq train- ing methods render the models shorthanded while attempting to bridge the gap between these neighbouring contexts. In this paper, we tackle this problem by using infilling tech- niques involving prediction of missing steps in a narrative while generating textual descrip- tions from a sequence of images. We also present a new large scale visual procedure telling (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps us- ing infilling technique demonstrates the effec- tiveness in visual procedures with more co- herent texts. We conclusively show a ME- TEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual sto- rytelling. We also demonstrate the effects of interposing new text with missing images during inference. The code and the dataset will be publicly available at https://visual- narratives.github.io/Visual-Narratives. 1 Introduction Humans process information from their surround- ing contexts from multiple modalities. These sit- uated contexts are often derived from a modality (source) and expressed in another modality (target). Recent advances have seen a surge of interest in vision and language as source and target modalities respectively. One such widely studied task is image captioning (Hossain et al., 2019; Liu et al., 2019) which provides a textual description T given an im- age I. In contrast, visual storytelling (Huang et al., 2016) is the task of generating a sequence of textual �1 ResNet-152 Narrative Context Layer Visual Feature Extraction First cream the butter and the vanilla extract together with a hand mixer. It only takes a few minutes . Scrape the sides of the bowl with a spatula as needed . Then give it a mix again for about 15 seconds . Next add in your powdered sugar little by little , with the mixer on low , until all of the powdered sugar is blended in . Next you can add a little bit of milk at a time to get the right consistency that you want . To use a piping back fold it over one of your hands , and open up the middle , or you can fold it over a tall glass . Figure 1: Overview of infilling in visual procedures. Image in the second step is masked while the model generates the corresponding textual description from surrounding context. descriptions ({T1, T2, ..., Tn}) from a sequence of images ({I1, I2, ..., In}). A fundamental incongruity between how …

Content truncated to 500 words