CapOnImage: Context-driven Dense-Captioning on Image

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3449 - 3465 December 7-11, 2022 ©2022 Association for Computational Linguistics CapOnImage: Context-driver Dense-Captioning On Image Yiqi Gao1∗, Xinglin Hou2, Yuanmeng Zhang2, Tiezheng Ge2 Yuning Jiang2, Peng Wang1† 1School of Computer Science, Northwestern Polytechnical University 2Alibaba Group 1gyqjz@mail.nwpu.edu.cn, 1peng.wang@nwpu.edu.cn 2{xinglin.hxl, zhangyuanmeng.zym, tiezheng.gtz, mengzhu.jyn}@alibaba-inc.com Abstract Existing image captioning systems are dedi- cated to generating narrative captions for im- ages, which are spatially detached from the image in presentation. However, texts can also be used as decorations on the image to high- light the key points and increase the attractive- ness of images. In this work, we introduce a new task called captioning on image (CapOn- Image)1, which aims to generate dense captions at different locations of the image based on con- textual information. For this new task, we in- troduce a large-scale benchmark called CapOn- Image2M, which contains 2.1 million product images, each with an average of 4.8 spatially lo- calized captions. To fully exploit the surround- ing visual context to generate the most suitable caption for each location, we propose a multi- modal pre-training model with multi-level pre- training tasks that progressively learn the corre- spondence between texts and image locations from easy to hard. To avoid generating redun- dant captions for nearby locations, we further enhance the location embedding with neighbor locations . Compared with other image cap- tioning model variants, our model achieves the best results in both captioning accuracy and diversity aspects. 1 Introduction Building upon the advances in computer vision and natural language processing areas, the new research direction called vision-and-language has attracted more and more attentions, which pushes to tackle new problems that need to bridge the two areas to advance the concept comprehension and reasoning capabilities. The image captioning task, as one of the most classic vision-and-language tasks, aims to generate natural language descriptions for images (Vinyals ∗ Work done during an internship at Alibaba Group. 1† Corresponding author. Code and dataset will be released at https://github. com/YqGao716/CapOnImage a row of building connected together. a brick walk way. a yellow building with a red roof. (b) e-commercial product image (c) social platform picture (a) dense captioning on visual genome dataset (soft, high elasticity nylon fiber) (Makeup in 10s) (Uniform and precise effect) (Novice friendly) (Genuine and discounted) (Add to favorite and delivery today) Figure 1: Illustration of descriptive texts on image sce- narios. The captions and images in existing tasks (e.g visual genome (a) (Krishna et al., 2017b)) are spatially detached, without any mutual association. In contrast, our CapOnImage task aims to generate descriptive text on the image, which has strong necessity and broad ap- plication prospects in some scenarios ((b) and (c)). et al., 2015; Anderson et al., 2018; Zhang et al., 2021; Johnson et al., 2016). However, the captions and the images in this task are spatially detached in presentation, without any association between each other (Figure 1(a)). In fact, there are many scenar- ios where the image and …

Content truncated to 500 words