Control Image Captioning Spatially and Temporally

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2014–2025 August 1–6, 2021. ©2021 Association for Computational Linguistics 2014 Control Image Captioning Spatially and Temporally Kun Yan†∗, Lei Ji ‡§¶, Huaishao Luo∥, Ming Zhou¶, Nan Duan¶, Shuai Ma† †SKLSDE Lab, Beihang University, Beijing, China ‡Institute of Computing Technology, CAS, Beijing, China §University of Chinese Academy of Sciences, Beijing, China ¶Microsoft Research Asia, Beijing, China ∥Southwest Jiaotong University, Chengdu, China †{kunyan,mashuai}@buaa.edu.cn ¶{leiji,mingzhou,nanduan}@microsoft.com ∥huaishaoluo@gmail.com Abstract Generating image captions with user inten- tion is an emerging need. The recently published Localized Narratives dataset takes mouse traces as another input to the image cap- tioning task, which is an intuitive and efficient way for a user to control what to describe in the image. However, how to effectively em- ploy traces to improve generation quality and controllability is still under exploration. This paper aims to solve this problem by propos- ing a novel model called LoopCAG, which connects Contrastive constraints and Attention Guidance in a Loop manner, engaged explicit spatial and temporal constraints to the gener- ating process. Precisely, each generated sen- tence is temporally aligned to the correspond- ing trace sequence through a contrastive learn- ing strategy. Besides, each generated text to- ken is supervised to attend to the correct visual objects under heuristic spatial attention guid- ance. Comprehensive experimental results demonstrate that our LoopCAG model learns better correspondence among the three modal- ities(vision, language, and traces) and achieves SOTA performance on trace controlled image captioning task. Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process. 1 Introduction Image captioning is a fundamental task to examine whether an intelligent system can understand the visual world by letting the system describe it with natural language. Generating a reasonable caption requires the model to link linguistic tokens to ob- jects, relationships, scenes of the visual world in the input image. Thus, a great captioning model will help us better understand what characteristics promise a good joint visual-linguistic representa- tion. ∗Contribution during internship at MSRA. In this picture there is a stand on a ground. On the backside there is a person. He is riding on a horse. He is wearing a cap. He is in between the fence. There is a flags on a wall. On the left side there is a score board on a table and flower plants. We can see in the background sky and trees. Figure 1: A showcase of Trace Controlled Image Cap- tion. Given an image together with a mouse trace rep- resenting user intention, the task is to generate the cor- responding captions aligned with each part of the trace. In this case, the trace and the caption marked with the same color correspond to each other. Most previous attempts aim to describe the im- age indicating the salient objects and relations with- out considering user intention. To generate con- trollable and explainable captions, …

Content truncated to 500 words