Are Scene Graphs Good Enough to Improve Image Captioning?

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 504–515 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 504 Are scene graphs good enough to improve Image Captioning? Victor Milewski1 Marie-Francine Moens1 Iacer Calixto2,3 1KU Leuven 2New York University 3ILLC, University of Amsterdam {victor.milewski,sien.moens}@cs.kuleuven.be iacer.calixto@nyu.edu Abstract Many top-performing image captioning mod- els rely solely on object features computed with an object detection model to generate image descriptions. However, recent studies propose to directly use scene graphs to intro- duce information about object relations into captioning, hoping to better describe interac- tions between objects. In this work, we thor- oughly investigate the use of scene graphs in image captioning. We empirically study whether using additional scene graph encoders can lead to better image descriptions and pro- pose a conditional graph attention network (C- GAT), where the image captioning decoder state is used to condition the graph updates. Finally, we determine to what extent noise in the predicted scene graphs influence caption quality. Overall, we find no significant dif- ference between models that use scene graph features and models that only use object detec- tion features across different captioning met- rics, which suggests that existing scene graph generation models are still too noisy to be use- ful in image captioning. Moreover, although the quality of predicted scene graphs is very low in general, when using high quality scene graphs we obtain gains of up to 3.3 CIDEr compared to a strong Bottom-Up Top-Down baseline.1 1 Introduction Scene understanding is a complex and intricate ac- tivity which humans perform effortlessly but that computational models still struggle with. An impor- tant backbone of scene understanding is being able to detect objects and relations between objects in an image, and scene graphs (Johnson et al., 2015; Anderson et al., 2016) are a closely related data 1We open source the codebase to reproduce all our ex- periments in https://github.com/iacercalixto/ butd-image-captioning. structure that explicitly annotates an image with its objects and relations in context. Scene graphs can be used to improve important visual tasks that require scene understanding, e.g. image indexing and search (Johnson et al., 2015) or scene construc- tion and generation (Johnson et al., 2017, 2018), and there is evidence that they can also be used to improve image captioning (Yang et al., 2019; Li and Jiang, 2019). However, the de facto standard in top-performing image captioning models to date use strong object features only, e.g. obtained with a pretrained Faster R-CNN (Ren et al., 2015), and no explicit relation information (Anderson et al., 2018; Lu et al., 2018; Yu et al., 2019a). One possible explanation to this observation is that by using detected objects we already capture the more important information that characterises a scene, and that relation information is already implicitly learned in such models. Another expla- nation is that relations are simply not as important as we hypothesise and that we gain no …

Content truncated to 500 words