Compositional Generalization in Image Captioning

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 87–98 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 87 Compositional Generalization in Image Captioning Mitja Nikolaus University of T¨ubingen∗ mitja.nikolaus@posteo.de Mostafa Abdou University of Copenhagen abdou@di.ku.dk Matthew Lamm Stanford University mlamm@cs.stanford.edu Rahul Aralikatte University of Copenhagen rahul@di.ku.dk Desmond Elliott University of Copenhagen de@di.ku.dk Abstract Image captioning models are usually evaluated on their ability to describe a held-out set of im- ages, not on their ability to generalize to un- seen concepts. We study the problem of com- positional generalization, which measures how well a model composes unseen combinations of concepts when describing images. State- of-the-art image captioning models show poor generalization performance on this task. We propose a multi-task model to address the poor performance, that combines caption genera- tion and image–sentence ranking, and uses a decoding mechanism that re-ranks the cap- tions according their similarity to the image. This model is substantially better at generaliz- ing to unseen combinations of concepts com- pared to state-of-the-art captioning models. 1 Introduction When describing scenes, humans are able to almost arbitrarily combine concepts, producing novel combinations that they have not previously observed (Matthei, 1982; Piantadosi and Aslin, 2016). Imagine encountering a purple-colored dog in your town, for instance. Given that you under- stand the concepts PURPLE and DOG, you are able to compose them together to describe the dog in front of you, despite never having seen one before. Image captioning models attempt to auto- matically describe scenes in natural language (Bernardi et al., 2016). Most recent approaches generate captions using a recurrent neural net- work, where the image is represented by features extracted from a Convolutional Neural Network (CNN). Although state-of-the-art models show good performance on challenge datasets, as mea- sured by text-similarity metrics, their performance ∗The work was carried out during a visit to the University of Copenhagen. A white cat sitting on a laptop computer A white dog running along a beach A big brown dog sitting on a couch Training Evaluation White things Dogs Figure 1: We evaluate whether image captioning mod- els are able to compositionally generalize to unseen combinations of adjectives, nouns, and verbs by forc- ing paradigmatic gaps in the training data. as measured by human judges is low when com- pared to human-written captions (Vinyals et al., 2017, Section 5.3.2). It is widely believed that systematic compo- sitionality is a key property of human language that is essential for making generalizations from limited data (Montague, 1974; Partee, 1984; Lake et al., 2017). In this work, we investigate to what extent image captioning models are capable of compositional language understanding. We ex- plore whether these models can compositionally generalize to unseen adjective–noun and noun– verb composition pairs, in which the constituents of the pair are observed during training but the combination is not, thus introducing a paradig- matic gap in the training data, as illustrated in Fig- ure 1. We define new training and evaluation splits of the COCO dataset …

Content truncated to 500 words