Informative Image Captioning with External Sources of Information

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6485–6494 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6485 Informative Image Captioning with External Sources of Information Sanqiang Zhao University of Pittsburgh sanqiang.zhao@pitt.edu Piyush Sharma Tomer Levinboim Radu Soricut Google Research, Venice, CA 90291 {piyushsharma,tomerl,rsoricut}@google.com Abstract An image caption should fluently present the essential information in a given image, includ- ing informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usu- ally trained to generate captions that only con- tain common object names, thus falling short on an important “informativeness” dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the im- age in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the out- put, resulting in captions that are both fluent and informative. 1 Introduction Much of the visual information available on the web is in the form of billions of images, but that information is not readily accessible to those with visual impairments, or those with slow in- ternet speeds. Automatic image captioning can help alleviate this problem, but its usefulness is directly proportional to how much information an automatically-produced caption can convey. As it happens, the goal of learning good models for im- age captioning (in terms of generalization power) is at odds with the goal of producing highly in- formative captions (in terms of fine-grained en- tity mentions). For this reason, previous ap- proaches to learning image captioning models at web scale (Sharma et al., 2018) had to compro- mise on the informativeness aspect, and trained models that could not produce fine-grained entity mentions (e.g., “Season of the Witch”) and instead Baseline Model: “return to the main poster page for film” Our Approach: “extra large movie poster image for Season of the Witch” Baseline Model: “folk rock artist per- forms on stage during festival” Our Approach: “Eric Clapton per- forms on stage during the Crossroads Guitar Festival” Figure 1: Generating informative captions using fine- grained entity information from external sources; base- line outputs from Sharma et al. (2018). settled for the conceptual (i.e., hypernym) variant for such entities (e.g., “film”) as shown in Fig. 1. We present an approach that solves this problem by leveraging upstream models that are capable of producing fine-grained entity names, and integrat- ing them in a controlled manner to produce cap- tions that are both fluent and highly informative. The standard approach to the image caption- ing task uses ⟨image, caption⟩ pairs to train an image-to-text encoder-decoder model. The “im- age encoder” is usually a Convolutional Neural Network that extracts image features. The “text decoder” is usually a Recurrent Neural Network or a Transformer Network (Vaswani …

Content truncated to 500 words