Transforming Visual Scene Graphs to Image Captions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12427–12440 July 9-14, 2023 ©2023 Association for Computational Linguistics Transforming Visual Scene Graphs to Image Captions Xu Yang1 Jiawei Peng1 Zihua Wang1 Haiyang Xu2∗ Qinghao Ye 2 Chenliang Li2 Songfang Huang2 Fei Huang2 Zhangzikang Li1 Yu Zhang1∗ 1 School of Computer Science & Engineering, Key Lab of Computer Network & Information Integration (Ministry of Education), Southeast Univ., Nanjing, China 2Alibaba Group {101013120, pengjiawei, zihua, zhang_yu}@seu.edu.cn,{shuofeng.xhy, yeqinghao.yqh, lcl193798, songfang.hsf}@alibaba-inc.com, {fairhuang, lizhangzikang}@gmail.com Abstract We propose to TransForm Scene Graphs into more descriptive Captions (TFSGC). In TF- SGC, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embed- ding, different graph embeddings contain di- verse specific knowledge for generating the words with different part-of-speech, e.g., ob- ject/attribute embedding is good for generat- ing nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based de- coder, where each expert is built on MHA, for discriminating the graph embeddings to gener- ate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a simple and homogeneous encoder-decoder unlike the pre- vious heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture en- ables us to unify the training configuration of the whole model instead of specifying differ- ent training strategies for diverse sub-networks as in the heterogeneous pipeline, which re- leases the training difficulty. Extensive experi- ments on the MS-COCO captioning benchmark validate the effectiveness of our TFSGC. The code is in: https://anonymous.4open.science/r/ ACL23_TFSGC. 1 Introduction Image captioning, which aims to generate one sen- tence for describing multi-aspects of an image, has made huge progress since the proposal of the encoder-decoder framework (Vinyals et al., 2015; Xu et al., 2015). Such a framework contains one visual encoder to extract a series of visual features from the image and one language decoder to gen- erate captions from the extracted visual features. Since the visual encoder is usually well pre-trained by image classification and object detection, the *Corresponding authors. extracted features contain abundant knowledge of the object categories, which enables the captioning model to generate object-abundant captions. However, object category is not the only visual pattern that matters for high-quality captions (An- derson et al., 2018; Jiang et al., 2020), object at- tributes and relations also play significant roles in generating descriptive captions, i.e., the caption containing multi-aspects of an image. Motivated by this, researchers propose to incorporate addi- tional semantic knowledge, e.g., object categories, attributes, and relations, into the captioning models by using the scene graph as the mediator (Yao et al., 2018; Yang et al., 2020). Scene graphs assign each object node with certain attribute nodes and some pairwise objects with certain relation nodes. These nodes are represented by the corresponding seman- tic tags, e.g., as shown in Fig. 1, the object “dog” is assigned with the attribute “black” and the pairwise objects “dog” and “fish” have …

Content truncated to 500 words