Visual Storytelling

Visual Storytelling Ting-Hao (Kenneth) Huang1∗, Francis Ferraro2∗, Nasrin Mostafazadeh3, Ishan Misra1, Aishwarya Agrawal4, Jacob Devlin6, Ross Girshick5, Xiaodong He6, Pushmeet Kohli6, Dhruv Batra4, C. Lawrence Zitnick5, Devi Parikh4, Lucy Vanderwende6, Michel Galley6, Margaret Mitchell6,7 Microsoft Research 1 Carnegie Mellon University; tinghaoh@cs.cmu.edu 2 Johns Hopkins University; ferraro@cs.jhu.edu 3 University of Rochester 4 Virginia Tech 6 Microsoft Research; {jdevlin,lucyv,mgalley}@microsoft.com 5 Facebook AI Research, 7 Now at Google Research; margarmitchell@gmail.com Abstract We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling. The first release of this dataset, SIND1 v.1, includes 81,743 unique photos in 20,211 se- quences, aligned to both descriptive (caption) and story language. We establish several strong baselines for the storytelling task, and motivate an automatic metric to benchmark progress. Modelling concrete description as well as figurative and social language, as pro- vided in this dataset and the storytelling task, has the potential to move artificial intelligence from basic understandings of typical visual scenes towards more and more human-like un- derstanding of grounded event structure and subjective expression. 1 Introduction Beyond understanding simple objects and concrete scenes lies interpreting causal structure; making sense of visual input to tie disparate moments to- gether as they give rise to a cohesive narrative of events through time. This requires moving from rea- soning about single images – static moments, de- void of context – to sequences of images that depict events as they occur and change. On the vision side, progressing from single images to images in context allows us to begin to create an artificial intelligence (AI) that can reason about a visual moment given ∗T.H. and F.F. contributed equally to this work. 1Sequential Images Narrative Dataset. The nal release was made available through Microsoft.origi- Re- lated future releases for Visual Storytelling (VIST) are on visionandlanguage.net/VIST. A group of people that are sitting next to each other. Adult male wearing sunglasses lying down on black pavement. The sun is setting over the ocean and mountains. Having a good time bonding and talking. [M] got exhausted by the heat. Sky illuminated with a brilliance of gold and orange hues. DII SIS Figure 1: Example language difference between descrip- tions for images in isolation (DII) vs. stories for images in sequence (SIS). what it has already seen. On the language side, pro- gressing from literal description to narrative helps to learn more evaluative, conversational, and abstract language. This is the difference between, for ex- ample, “sitting next to each other” versus “having a good time”, or “sun is setting” versus “sky illumi- nated with a brilliance...” (see Figure 1). The first descriptions capture image content that is literal and concrete; the second requires further inference about what a good time may look like, or what is special and worth sharing about a particular sunset. We introduce the first dataset of sequential im- ages with corresponding descriptions, which cap- tures some of these subtle but important differ- ences, and advance the task of visual …

Content truncated to 500 words