SANDI: Story-and-Images Alignment

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 989–999 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 989 SANDI: Story-and-Images Alignment Sreyasi Nag Chowdhury Simon Razniewski Gerhard Weikum Max Planck Institute for Informatics, Saarbrücken, Germany (sreyasi, srazniew, weikum)@mpi-inf.mpg.de Abstract The Internet contains a multitude of social me- dia posts and other stories where text is inter- spersed with images. In these contexts, im- ages are not simply used for general illustra- tion, but are judiciously placed in certain spots of a story for multimodal descriptions and nar- ration. In this work we analyze the problem of text-image alignment, and present SANDI, a methodology for automatically selecting im- ages from an image collection and aligning them with text paragraphs of a story. SANDI combines visual tags, user-provided tags and background knowledge, and uses an Integer Linear Program to compute alignments that are semantically meaningful. Experiments show that SANDI can select and align images with texts with high quality of semantic fit. 1 Introduction It is well-known (and supported by studies Lester (2013); Messaris and Abraham (2001)) that the most powerful messages are delivered with a com- bination of words and pictures. On the Internet, such multimodal content is abundant in the form of news articles, social media posts, and personal blog posts where authors enrich their stories with care- fully chosen and placed images. As an example, consider a vacation report, to be posted on a blog site or online community. The backbone of the travel report is a textual narration, but the user typi- cally places illustrative images in appropriate spots, carefully selected from her photo collection from this trip. These images can either show specific objective highlights such as waterfalls, mountain hikes or animal encounters, or may serve to depict the thematic mood of the trip, e.g., by showing nice sunsets. Another example is brochures for various organizations. Here, the text describes the mission, achievements and ongoing projects, and is accom- panied with judiciously selected and placed photos Figure 1: The story-and-images alignment problem. of buildings, people, products and images depicting the subjects of interest, e.g., galaxies or telescopes for research in astrophysics. The generation of such multimodal stories re- quires substantial human judgement and reasoning, and is thus time-consuming and labor-intensive. In particular, the effort on the human side includes selecting the right images from a pool of story- specific photos (e.g., the traveler’s own photos) and possibly also from a broader pool for visual illustra- tion (e.g., from Pinterest). Even if the set of photos were exactly given, there is still considerable effort to place them within appropriate paragraphs, pay- ing attention to the semantic coherence between surrounding text and image. In this paper, we set out to automate this human task, formalizing it as a Story-AND-Images (SANDI) alignment problem. Problem Statement. Given a story-like text docu- ment and a set of images, the problem is to automat- ically decide where individual images are placed in the …

Content truncated to 500 words