Pragmatic Issue-Sensitive Image Captioning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1924–1938 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1924 Pragmatic Issue-Sensitive Image Captioning Allen Nie1 Reuben Cohn-Gordon2 1Department of Computer Science 2Department of Linguistics Stanford University {anie, reubencg, cgpotts}@stanford.edu Christopher Potts1,2 Abstract Image captioning systems need to produce texts that are not only true but also relevant in that they are properly aligned with the cur- rent issues. For instance, in a newspaper arti- cle about a sports event, a caption that not only identifies the player in a picture but also com- ments on their ethnicity could create unwanted reader reactions. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, the captioner is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is rele- vant. For the sports article, we could construct a partition that places images into equivalence classes based on player position. To model this task, we use an extension of the Ratio- nal Speech Acts model. Our extension is built on top of state-of-the-art pretrained neural im- age captioners and explicitly uses image par- titions to control caption generation. In both automatic and human evaluations, we show that these models generate captions that are de- scriptive and issue-sensitive. Finally, we show how ISIC can complement and enrich the re- lated task of Visual Question Answering. 1 Introduction Image captioning systems have improved dramat- ically over the last few years (Karpathy and Fei- Fei, 2015; Vinyals et al., 2015; Hendricks et al., 2016; Rennie et al., 2017; Anderson et al., 2018), creating new opportunities to design systems that are not just accurate, but also produce descrip- tions that include relevant, characterizing aspects of their inputs. Many of these efforts are guided by the insight that high-quality captions are implicitly shaped by the communicative goal of identifying the target image up to some level of granularity (Vedantam et al., 2017; Mao et al., 2016; Luo et al., 2018; Cohn-Gordon et al., 2018). In this paper, we seek to more tightly control the Issues Target Caption a small brown bird with a tan chest and a tan beak this bird has a brown crown a white eyebrow and a rounded belly What is the color of the bird? What is the head pattern of the bird? Figure 1: Examples highlighting the power of an issue- sensitive image captioner. Four images are partitioned in two ways, each capturing different issues by group- ing them into equivalence classes. The first row con- trasts the brown and grey color of the bird, and the sec- ond contrasts the existence of white eyebrows. The tar- get image is the same in both cases, but the partition leads to different captions that key into the structure of the input issue. information that a pretrained captioner includes in its output texts. Our focus is on generating captions that are relevant to the current issues. To see how important …

Content truncated to 500 words