Concadia: Towards Image-Based Text Generation with a Purpose

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4667 - 4684 December 7-11, 2022 ©2022 Association for Computational Linguistics Concadia: Towards Image-Based Text Generation with a Purpose Elisa Kreiss1 Fei Fang2 Noah D. Goodman2,3 Christopher Potts1 1Department of Linguistics 2Department of Computer Science 3Department of Psychology Stanford University Stanford, CA 94305 USA {ekreiss, feifang, ngoodman, cgpotts}@stanford.edu Abstract Current deep learning models often achieve excellent results on benchmark image-to-text datasets but fail to generate texts that are useful in practice. We argue that to close this gap, it is vital to distinguish descriptions from cap- tions based on their distinct communicative roles. Descriptions focus on visual features and are meant to replace an image (often to increase accessibility), whereas captions ap- pear alongside an image to supply additional information. To motivate this distinction and help people put it into practice, we introduce the publicly available Wikipedia-based dataset Concadia consisting of 96,918 images with corresponding English-language descriptions, captions, and surrounding context. Using in- sights from Concadia, models trained on it, and a preregistered human-subjects experiment with human- and model-generated texts, we characterize the commonalities and differences between descriptions and captions. In addition, we show that, for generating both descriptions and captions, it is useful to augment image-to- text models with representations of the textual context in which the image appeared. 1 Introduction Image-based natural language generation (NLG) has great potential for productive applications in accessibility, image search, creative writing, and navigational instruction tasks, among other areas. However, current systems fall short of realizing this potential even though they can generally produce fluent, truthful texts in a wide range of scenarios (e.g., Dognin et al., 2022; Guinness et al., 2018; Gurari et al., 2020). In this paper, we identify and address one of the fundamental limitations holding these systems back: insufficient attention to the communicative purpose of the text being generated. We focus on the high-level distinction between descriptions and captions. Descriptions are created for the purpose of replacing an image in context, Figure 1: An example from our Wikipedia-based corpus Concadia of an image from the article on bananas with two associated texts: the description as provided in the image’s alt text, and the caption as displayed below the image in the article. For captions, the image content is presupposed whereas descriptions aim to stand in for the image. While grounded in the same image, description and caption convey vastly different information. most notably used to make an image accessible to users who can’t see them. Captions, in contrast, presuppose access to the image and seek to provide additional information. For example, in Figure 1, the description summarizes the scene and describes the features that are clearly visible in the image, whereas the caption provides almost no visual de- tails and instead highlights the relevance of this image to the article. Both texts are grounded in the same image and have clear purposes, but neither can play the role of the other. To enable in-depth exploration of …

Content truncated to 500 words