Cross-modal Coherence Modeling for Caption Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6525–6535 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6525 Clue: Cross-modal Coherence Modeling for Caption Generation Malihe Alikhani Rutgers University malihe.alikhani@rutgers.edu Piyush Sharma Google Research piyushsharma@google.com Shengjie Li Rutgers University sl1560@rutgers.edu Radu Soricut Google Research rsoricut@google.com Matthew Stone Rutgers University mdstone@rutgers.edu Abstract We use coherence relations inspired by compu- tational models of discourse to study the infor- mation needs and goals of image captioning. Using an annotation protocol specifically de- vised for capturing image–caption coherence relations, we annotate 10,000 instances from publicly-available image–caption pairs. We in- troduce a new task for learning inferences in imagery and text, coherence relation predic- tion, and show that these coherence annota- tions can be exploited to learn relation clas- sifiers as an intermediary step, and also train coherence-aware, controllable image caption- ing models. The results show a dramatic im- provement in the consistency and quality of the generated captions with respect to informa- tion needs specified via coherence relations. 1 Introduction The task of image captioning is seemingly straight- forward to define: use natural language to generate a description that captures the salient content of an image. Initial datasets, such as MSCOCO (Lin et al., 2014) and Flickr (Young et al., 2014), ap- proached this task directly, by asking crowd work- ers to describe images in text. Unfortunately, such dedicated annotation efforts cannot yield enough data for training robust generation models; the re- sulting generated captions are plagued by content hallucinations (Rohrbach et al., 2018; Sharma et al., 2018) that effectively preclude them for being used in real-world applications. In introducing the Conceptual Captions dataset, Sharma et al. (2018) show that this dataset is large enough, at 3.3M examples, to significantly allevi- ate content hallucination. However, because the technique for creating such a large-scale resource relies on harvesting existing data from the web, it no longer guarantees consistent image–text rela- tions. For example, along with descriptive captions Figure 1: Output of a coherence-aware model for vari- ous coherence relations. Content that establishes the in- tended relation is underlined. (Photo credit: Blue Des- tiny / Alamy Stock Photo) Visible: horse and rider jumping a fence. Meta: horse and rider jumping a fence during a race. Subjective: the most beautiful horse in the world. Story: horse competes in the event. (e.g.,“this is a person in a suit”), the dataset also includes texts that provide contextual background (e.g., “this is the new general manger of the team”) and subjective evaluations (e.g., “this is stylish”). As a result, current captioning models trained on Conceptual Captions avoid content hallucination but also introduce different, more subtle and harder- to-detect issues related to possible context halluci- nations (i.e., is this actually the new general man- ager?) or subjective-judgement hallucinations (i.e., whose judgment is this anyway?). In this paper, we propose to tackle this issue of large-scale image-caption consistency using a coherence-aware approach inspired by the frame- work of discourse coherence theory (Hobbs, 1978; Phillips, 1977). …

Content truncated to 500 words