Sort Story: Sorting Jumbled Images and Captions into Stories

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 925–931, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Sort Story: Sorting Jumbled Images and Captions into Stories Harsh Agrawal∗,1 Arjun Chandrasekaran∗,1,† Dhruv Batra3,1 Devi Parikh3,1 Mohit Bansal4,2 1Virginia Tech 2TTI-Chicago 3Georgia Institute of Technology 4UNC Chapel Hill {harsh92, carjun, dbatra, parikh}@vt.edu, mbansal@cs.unc.edu Abstract Temporal common sense has applications in AI tasks such as QA, multi-document summa- rization, and human-AI communication. We propose the task of sequencing – given a jum- bled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coher- ent story. We present multiple approaches, via unary (position) and pairwise (order) pre- dictions, and their ensemble-based combina- tions, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense. 1 Introduction Sequencing is a task for children that is aimed at im- proving understanding of the temporal occurrence of a sequence of events. The task is, given a jumbled set of images (and maybe captions) that belong to a single story, sort them into the correct order so that they form a coherent story. Our motivation in this work is to enable AI systems to better under- stand and predict the temporal nature of events in the world. To this end, we train machine learning models to perform the task of “sequencing”. Temporal reasoning has a number of applications such as multi-document summarization of multiple sources of, say, news information where the relative order of events can be useful to accurately merge information in a temporally consistent manner. In question answering tasks (Richardson et al., 2013; ∗Denotes equal contribution. †Part of this work was done during an internship at TTIC. Figure 1: (a) The input is a jumbled set of aligned image-caption pairs. (b) Actual output of the system – an ordered sequence of image-caption pairs that form a coherent story. Fader et al., 2014; Weston et al., 2015; Ren et al., 2015), answering questions related to when an event occurs, or what events occurred prior to a particular event require temporal reasoning. A good temporal model of events in everyday life, i.e., a “temporal common sense”, could also improve the quality of communication between AI systems and humans. Stories are a form of narrative sequences that have an inherent temporal common sense structure. We propose the use of visual stories depicting personal events to learn temporal common sense. We use stories from the Sequential Image Narrative Dataset (SIND) (Ting-Hao Huang, 2016) in which a set of 5 aligned image-caption pairs together form a co- herent story. Given an input story that is jumbled (Fig. 1(a)), we train machine learning models to sort them into a coherent story (Fig. 1(b)).1 1Note that ‘jumbled’ here refers to the loss of temporal or- dering; image-caption pairs are still …

Content truncated to 500 words