Understanding Guided Image Captioning Performance across Domains

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 183–193 November 10–11, 2021. ©2021 Association for Computational Linguistics 183 Understanding Guided Image Captioning Performance across Domains Edwin G. Ng∗ Bo Pang Piyush Sharma Radu Soricut Google Research eg.ng@alum.utoronto.ca, {bopang,piyushsharma,rsoricut}@google.com Abstract Image captioning models generally lack the ca- pability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and in- formation overload. We present a Transformer- based model with the ability to produce cap- tions focused on specific objects, concepts or actions in an image by providing them as guid- ing text to the model. Further, we evalu- ate the quality of these guided captions when trained on Conceptual Captions which contain 3.3M image-level captions compared to Visual Genome which contain 3.6M object-level cap- tions. Counter-intuitively, we find that guided captions produced by the model trained on Conceptual Captions generalize better on out- of-domain data. Our human-evaluation results indicate that attempting in-the-wild guided image captioning requires access to large, unrestricted-domain training datasets, and that increased style diversity (even without increas- ing the number of unique tokens) is a key fac- tor for improved performance. 1 Introduction Describing the content of an image using natural language is generically referred to as image cap- tioning, but there are a variety of ways in which this can be achieved: by focusing on the most salient aspects of an image, as in MSCOCO (Lin et al., 2014) or Conceptual Captions (Sharma et al., 2018); on most of the groundable concepts in an image, as in Image Paragraphs (Krause et al., 2017) or Localized Narratives (Pont-Tuset et al., 2020); or on a predefined set of objects, as in dense captioning (Johnson et al., 2016). These various approaches acknowledge that a typical real- world image may contain a varying number of ob- jects/concepts/actions that may be of interest to the caption consumer, and therefore the optimal description depends on the degree to which the ∗Work done as a part of the Google AI Residency. Figure 1: An illustration comparing the difference be- tween an image captioning model and a guided image captioning model. The guided image captioning model generates captions that focus on a specific object, con- cept or action of the image, provided to the model as free-form guiding text. caption covers what the user is interested in at any given moment. However, the vast majority of captioning solu- tions lack the ability to take into account user in- terest, and usually default to global descriptions. Such a description might say, for example “people attending a birthday party”, which may not satisfy a user interested in a description of “the cake”. In this paper, we capture user interest via the guid- ing text, a free-form text input that is assumed to be related to some concept(s) in the image; and we consider the Guided Image Captioning task, where the guiding text is provided to the model as an additional input to control the concepts …

Content truncated to 500 words