Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6514–6524 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6514 Bridging by Word: Image-Grounded Vocabulary Construction for Visual Captioning Zhihao Fan1, Zhongyu Wei1∗, Siyuan Wang1, Xuanjing Huang2 1School of Data Science, Fudan University, China 2School of Computer Science, Fudan University, China {fanzh18,zywei,wangsy18,xjhuang}@fudan.edu.cn Abstract Existing research for visual captioning usu- ally employs a CNN-RNN architecture that combines a CNN for image encoding with a RNN for caption generation, where the vo- cabulary is constructed from the entire train- ing dataset as the decoding space. Such ap- proaches typically suffer from the problem of generating N-grams which occur frequently in the training set but are irrelevant to the given image. To tackle this problem, we pro- pose to construct an image-grounded vocabu- lary that leverages image semantics for more effective caption generation. More concretely, a two-step approach is proposed to construct the vocabulary by incorporating both visual information and relationships among words. Two strategies are then explored to utilize the constructed vocabulary for caption generation. One constrains the generator to select words from the image-grounded vocabulary only and the other integrates the vocabulary informa- tion into the RNN cell during the caption gen- eration process. Experimental results on two public datasets show the effectiveness of our framework compared to state-of-the-art mod- els. Our code is available on Github1. 1 Introduction Recent years have witnessed growing popular- ity of research in multimodal learning across vi- sion and language. Image captioning (Xu et al., 2015), one of the most widely studied multimodal tasks, aims at constructing a short text descrip- tion given an image. Existing research on im- age captioning usually employs a CNN-RNN ar- chitecture with a Convolutional Neural Network (CNN) used for image feature extraction and a Re- ∗*Corresponding author 1https://github.com/LibertFan/ ImageCaption Figure 1: Two images from MS-COCO (Lin et al., 2014) with captions generated by NIC (Vinyals et al., 2015) and the corresponding ground truth (GT) cap- tions. Figure 2: Distribution of images in terms of the num- ber of distinct words used for their descriptions. X- axis: the number of distinct words in all correspond- ing ground truth captions v.s. Y-axis: the number of instances in MS-COCO. current Neural Network (RNN) for caption gener- ation (Vinyals et al., 2015). Although impressive results have been achieved, existing models suf- fer from the problem of generating N-grams which occurred frequently in the training set but are ir- relevant to the particular given image (Anderson et al., 2016; Dai et al., 2017). Examples of caption generation are shown in Figure 1. Two images are presented with model- generated captions (Vinyals et al., 2015) and their 6515 Figure 3: Overall framework of our proposed model. corresponding human-constructed ones. As we can see, the N-gram “a woman sitting at a ta- ble” is generated mistakenly for both images. This is because when generating a text sequence, the RNN-based generator tends to ignore the semantic meaning encoded in …

Content truncated to 500 words