Language-Driven Region Pointer Advancement for Controllable Image Captioning

Proceedings of the 28th International Conference on Computational Linguistics, pages 1922–1935 Barcelona, Spain (Online), December 8-13, 2020 1922 Language-Driven Region Pointer Advancement for Controllable Image Captioning Annika Lindh and Robert Ross and John D. Kelleher ADAPT Research Centre School of Computer Science, Technological University Dublin, Kevin Street, Dublin 8, IRELAND {annika.lindh;robert.ross;john.d.kelleher}@tudublin.ie Abstract Controllable Image Captioning is a recent sub-field in the multi-modal task of Image Captioning wherein constraints are placed on which regions in an image should be described in the generated natural language caption. This puts a stronger focus on producing more detailed descriptions, and opens the door for more end-user control over results. A vital component of the Controllable Image Captioning architecture is the mechanism that decides the timing of attending to each region through the advancement of a region pointer. In this paper, we propose a novel method for predicting the timing of region pointer advancement by treating the advancement step as a natural part of the language structure via a NEXT-token, motivated by a strong correlation to the sentence structure in the training data. We find that our timing agrees with the ground-truth timing in the Flickr30k Entities test data with a precision of 86.55% and a recall of 97.92%. Our model implementing this technique improves the state-of-the-art on standard captioning metrics while additionally demonstrating a considerably larger effective vocabulary size. 1 Introduction Image Captioning brings together the two fields of Computer Vision and Natural Language Generation into a task where the model needs to translate an input image into an appropriate natural language text description. The task leaves some ambiguity regarding which parts of the image should be mentioned and which ones can be excluded. This has led to a common problem where models tend to generate overly generic descriptions that seem to focus more on the category of the image than on its individual content (Devlin et al., 2015; Madhyastha et al., 2018). Recently, Cornia et al. (2019) introduced Controllable Image Captioning as a new sub-task of Image Captioning with a stronger focus on image details. In this task, the input to the model is an image along with bounding box coordinates for a sequence of regions (where each region can consist of one or more bounding boxes) that must be explicitly described in the candidate caption. Thus, in contrast to standard Image Captioning, a generic candidate caption would not meet the criteria of a suitable caption even if it contained no factual errors; this is reflected in the evaluation process where the candidate caption is only compared to those ground-captions that share the same sequence of regions. Fig. 1 shows an image and two corresponding captions from the Flickr30k dataset (Young et al., 2014) along with complementary data from Flickr30k Entities (Plummer et al., 2017) which provides annotations that link entities in the captions to region bounding boxes in the corresponding image. In terms of practical use, a Controllable Image Captioning model provides more flexibility and user- control over the generated captions, without …

Content truncated to 500 words