Improving Image Captioning with Better Use of Caption

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7454–7464 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7454 Improving Image Captioning with Better Use of Captions Zhan Shi∗, Xu Zhou†, Xipeng Qiu†, Xiaodan Zhu∗ ∗Ingenuity Labs Research Institute, Queen’s University ∗Department of Electrical and Computer Engineering, Queen’s University †School of Computer Science, Fudan University {z.shi,xiaodan.zhu}@queensu.ca,{16210240095,xpqiu}@fudan.edu.cn Abstract Image captioning is a multimodal problem that has drawn extensive attention in both the natural language processing and computer vi- sion community. In this paper, we present a novel image captioning architecture to bet- ter explore semantics available in captions and leverage that to enhance both image represen- tation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learn- ing. The representation is then enhanced with neighbouring and contextual nodes with their textual and visual features. During genera- tion, the model further incorporates visual rela- tionships using multi-task learning for jointly predicting word and object/predicate tag se- quences. We perform extensive experiments on the MSCOCO dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under a wide range of evaluation metrics. The code of our paper has been made publicly available. 1 1 Introduction Automatically generating a short description for a given image, a problem known as image cap- tioning (Chen et al., 2015), has drawn extensive attention in both the natural language processing and computer vision community. Inspired by the success of encoder-decoder frameworks with the attention mechanism, previous efforts on image captioning adopt variants of pre-trained convolu- tion neural networks (CNN) as the image encoder and recurrent neural networks (RNN) with visual attention as the decoder (Lu et al., 2017; Anderson et al., 2018; Xu et al., 2015; Lu et al., 2018). Many previous methods translate image repre- sentation into natural language sentences without 1 https://github.com/Gitsamshi/WeakVRD-Captioning Figure 1: Visual relationship graphs from a pre-trained detection model (Yao et al., 2018) (upper) and from the ground-truth caption (bottom). explicitly investigating semantic cues from texts and images. To remedy that, some research has also explored to detect high-level semantic con- cepts presented in images to improve caption gen- eration (Wu et al., 2016; Gan et al., 2017; You et al., 2016; Fang et al., 2015; Yao et al., 2017). It is believed by many that the inductive bias that leverages structured combination of concepts and visual relationships is of importance, which has led to better captioning models (Yao et al., 2018; Guo et al., 2019; Yang et al., 2019). These approaches obtain visual relationship graphs using models pre- trained from visual relationship detection (VRD) datasets, e.g., Visual Genome (Krishna et al., 2017), where the visual relationships capture semantics between pairs of localized objects connected by predicates, including spatial (e.g., cake-on-desk) and non-spatial semantic relationships (e.g., man- eat-food) (Lu et al., 2016). As in many other joint text-image modeling prob- lems, it is crucial to obtain a good semantic rep- resentation …

Content truncated to 500 words