Hierarchically-Attentive RNN for Album Summarization and Storytelling

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 966–971 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Hierarchically-Attentive RNN for Album Summarization and Storytelling Licheng Yu and Mohit Bansal and Tamara L. Berg UNC Chapel Hill {licheng, mbansal, tlberg}@cs.unc.edu Abstract We address the problem of end-to-end vi- sual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Sto- rytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines. 1 Introduction Since we first developed language, humans have always told stories. Fashioning a good story is an act of creativity and developing algorithms to replicate this has been a long running challenge. Adding pictures as input can provide information for guiding story construction by offering visual illustrations of the storyline. In the related task of image captioning, most methods try to generate descriptions only for individual images or for short videos depicting a single activity. Very recently, datasets have been introduced that extend this task to longer temporal sequences such as movies or photo albums (Rohrbach et al., 2016; Pan et al., 2016; Lu and Grauman, 2013; Huang et al., 2016). The type of data we consider in this paper pro- vides input illustrations for story generation in the form of photo albums, sampled over a few minutes to a few days of time. For this type of data, gen- erating textual descriptions involves telling a tem- porally consistent story about the depicted visual information, where stories must be coherent and take into account the temporal context of the im- ages. Applications of this include constructing vi- sual and textual summaries of albums, or even en- abling search through personal photo collections to find photos of life events. Previous visual storytelling works can be clas- sified into two types, vision-based and language- based, where image or language stories are con- structed respectively. Among the vision-based ap- proaches, unsupervised learning is commonly ap- plied: e.g., (Sigurdsson et al., 2016) learns the la- tent temporal dynamics given a large amount of albums, and (Kim and Xing, 2014) formulate the photo selection as a sparse time-varying directed graph. However, these visual summaries tend to be difficult to evaluate and selected photos may not agree with human selections. For language- based approaches, a sequence of natural language sentences are generated to describe a set of pho- tos. To drive this work (Park and Kim, 2015) col- lected a dataset mined from Blog Posts. However, this kind of data often contains contextual infor- mation or loosely related language. A more direct dataset was recently released (Huang et al., 2016), where multi-sentence stories are collected describ- ing photo albums via Amazon Mechanical Turk. …

Content truncated to 500 words