Geo-Aware Image Caption Generation

Proceedings of the 28th International Conference on Computational Linguistics, pages 3143–3156 Barcelona, Spain (Online), December 8-13, 2020 3143 Geo-Aware Image Caption Generation Sofia Nikiforova, Tejaswini Deoskar, Denis Paperno, Yoad Winter Utrecht University, Utrecht, the Netherlands {s.nikiforova, t.deoskar, d.paperno, y.winter}@uu.nl Abstract Standard image caption generation systems produce generic descriptions of images and do not utilize any contextual information or world knowledge. In particular, they are unable to generate captions that contain references to the geographic context of an image, for example, the loca- tion where a photograph is taken or relevant geographic objects around an image location. In this paper, we develop a geo-aware image caption generation system, which incorporates geo- graphic contextual information into a standard image captioning pipeline. We propose a way to build an image-specific representation of the geographic context and adapt the caption genera- tion network to produce appropriate geographic names in the image descriptions. We evaluate our system on a novel captioning dataset that contains contextualized captions and geographic metadata and achieve substantial improvements in BLEU, ROUGE, METEOR and CIDEr scores. We also introduce a new metric to assess generated geographic references directly and empiri- cally demonstrate our system’s ability to produce captions with relevant and factually accurate geographic referencing. 1 Introduction Image caption generation is a popular task that aims at producing a natural language description of a given image. A standard neural image captioning system consists of two stages: an “encoder”, a Convolutional Neural Network that encodes the visual features of an image as a vector, and a “decoder”, a language model that is initialized with this vector and generates a caption word by word. Recent research (Lu et al., 2018; Whitehead et al., 2018; Biten et al., 2019) has drawn attention to the fact that the standard approach is insufficient to imitate captions naturally produced by humans. People tend to describe images interpreting them based on context factors and world knowledge, while standard encoder-decoder captioning systems do not take any contextual or world knowledge into account. One of the aspects that are missing from standard caption generation systems is the ability to produce image descriptions influenced by the geographic context, i.e. geographic objects surrounding the image location. For example, consider the photograph in Figure 1: Figure 1: An example image. Ground Truth: A path through Pitshanger Park, near Ealing in the west London suburbs Automatically generated: a park bench sitting in the middle of a park This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 3144 The person who took this photograph1 captions it referring to the location of the photograph (“Pit- shanger Park”) and the most relevant parts of the surroundings (“Ealing”, “west London suburbs”). However, a neural caption generation system (Xu et al., 2015) merely describes the objects in the image (“park bench”, “park”). In this paper we present geo-aware image captioning, where geographic contextual information is incorporated into the generated captions. The contributions of this paper are as follows: • We compile a …

Content truncated to 500 words