The Role of Syntactic Planning in Compositional Image Captioning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 593–607 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 593 The Role of Syntactic Planning in Compositional Image Captioning Emanuele Bugliarello and Desmond Elliott Department of Computer Science University of Copenhagen {emanuele,de}@di.ku.dk Abstract Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more chal- lenging problem of generalizing to different distributions of images. Recently, Nikolaus et al. (2019) introduced a dataset to assess com- positional generalization in image captioning, where models are evaluated on their ability to describe images with unseen adjective–noun and noun–verb compositions. In this work, we investigate different methods to improve com- positional generalization by planning the syn- tactic structure of a caption. Our experiments show that jointly modeling tokens and syntac- tic tags enhances generalization in both RNN- and Transformer-based models, while also im- proving performance on standard metrics. 1 Introduction Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the con- tent of an image in natural language. To succeed in this task, a model first needs to recognize and un- derstand the properties of the image. Then, it needs to generate well-formed sentences, requiring both a syntactic and a semantic knowledge of the language (Hossain et al., 2019). Deep learning techniques are the standard approach to tackling this problem: images are represented by visual features extracted from Convolutional Neural Networks (e.g. He et al. 2016), and sentences are generated by condition- ing Recurrent Neural Networks (e.g. Hochreiter and Schmidhuber 1997), or Transformers (Vaswani et al., 2017) on the extracted visual features. While deep neural networks achieve impressive performance in a variety of applications, including image captioning, their ability to demonstrate com- positionality, defined as the algebraic potential to understand and produce novel combinations from known components (Loula et al., 2018), has been questioned. Semantic compositionality of language in neural networks has attracted interest in the com- munity (Irsoy and Cardie, 2014; Lake and Baroni, 2018; Baroni, 2019) as compositionality is conjec- tured to be a core feature not only of language but also of human thought (Fodor and Lepore, 2002). In image captioning, improving compositional generalization is a fundamental step towards gen- eralizable systems that can be employed in daily life. To this end, Nikolaus et al. (2019) recently introduced a compositional generalization dataset where models need to describe images that depict unseen compositions of primitive concepts. For ex- ample, models are trained to describe images with “white” entities and all types of “dog” concepts but never the adjective–noun composition of “white dog.” In their dataset, models are evaluated on their ability to caption images depicting the unseen com- position of held out concepts. Their study suggests that RNN-based captioning models do not com- positionally generalize, and that this is primarily attributable to the language generation component. In this paper, we study the potential for syntax to …

Content truncated to 500 words