Bridge the Gap: High-level Semantic Planning for Image Captioning

Proceedings of the 28th International Conference on Computational Linguistics, pages 3157–3167 Barcelona, Spain (Online), December 8-13, 2020 3157 Bridge the Gap: High-level Semantic Planning for Image Captioning Chenxi Yuan and Yang Bai Department of Computer Science and Technology Tsinghua University ycx18@mails.tsinghua.edu.cn bai-y18@mails.tsinghua.edu.cn Chun Yuan ∗ Tsinghua ShenZhen International Graduate School Tsinghua University Peng Cheng Laboratory yuanc@sz.tsinghua.edu.cn Abstract Recent image captioning models have made much progress for exploring the multi-modal in- teraction, such as attention mechanisms. Though these mechanisms can boost the interaction, there are still two gaps between the visual and language domains: (1) the gap between the visual features and textual semantics, (2) the gap between the disordering of visual features and the or- dering of texts. To bridge the gaps we propose a high-level semantic planning (HSP) mechanism that incorporates both a semantic reconstruction and an explicit order planning. We integrate the planning mechanism to the attention based caption model and propose the High-level Semantic PLanning based Attention Network (HS-PLAN). First, an attention based reconstruction mod- ule is designed to reconstruct the visual features with high-level semantic information. Then we apply a pointer network to serialize the features and obtain the explicit order plan to guide the generation. Experiments conducted on MS COCO show that our model outperforms previous methods and achieves the state-of-the-art performance of 133.4% CIDEr-D score. 1 Introduction Image captioning which aims to generate textual descriptions of images, is a significant task in both computer vision and natural language process. It not only requires recognizing and understanding the objects and attributes from the given image but also needs to verbalize them with natural language in proper order. Previous works with neural models follow the encoder-decoder paradigm that uses Convolutional Neu- ral Network (CNN) to encode the input image and apply Recurrent Neural Network (RNN) as decoder to generate the textual descriptions (as shown in Figure 1(a)) (Vinyals et al., 2015; Gan et al., 2017a; Chen et al., 2018; Gan et al., 2017b; Lu et al., 2017; Yang et al., 2016). To explore the multi-modal interaction between the visual content and textual description, some recent methods (Xu et al., 2015; Anderson et al., 2018) apply visual attention mechanism to model the interaction. The visual attention works by learning to selectively attend to image features extracted by the encoder when generating each word. For better interaction, a large number of works focus on boost the performance of neural models with improved attention mechanisms (Huang et al., 2019a; Huang et al., 2019b; Pan et al., 2020). However, there are still two gaps between the visual and language domains that visual attention does not address: (1) the gap between the visual features and textual semantics, (2) the gap between the disordering of visual fea- tures and the ordering of texts. For one thing, it is hard for the decoder to associate each word in the caption with the features without a high-level semantic understanding. For another thing, with the visual attention these neural models implicitly select which features to …

Content truncated to 500 words