Rˆ3Net:Relation-embedded Representation Reconstruction Network for Change Captioning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9319–9329 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9319 R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning Yunbin Tu1*, Liang Li2†, Chenggang Yan3, Shengxiang Gao1, Zhengtao Yu1† 1Yunnan Key Laboratory of Artificial Intelligence, Kunming University of Science and Technology 2 Key Lab of Intell. Info. Process., Inst. of Comput. Tech., Chinese Academy of Sciences 3 Intelligent Information Processing Laboratory, Hangzhou Dianzi University tuyunbin1995@foxmail.com, liang.li@ict.ac.cn Abstract Change captioning is to use a natural language sentence to describe the fine-grained disagree- ment between two similar images. Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the represen- tation of real change. In this paper, we pro- pose a Relation-embedded Representation Re- construction Network (R3Net) to explicitly dis- tinguish the real change from the large amount of clutter and irrelevant changes. Specifi- cally, a relation-embedded module is first de- vised to explore potential changed objects in the large amount of clutter. Then, based on the semantic similarities of corresponding lo- cations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and fur- ther model the difference representation. Be- sides, we introduce a syntactic skeleton predic- tor (SSP) to enhance the semantic interaction between change localization and caption gen- eration. Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets 1. 1 Introduction Change captioning aims to generate a natural lan- guage sentence to detail what has changed in a pair of similar images. It has many practical ap- plications, such as assisted surveillance, medical imaging, and computer assisted tracking of changes in media assets (Jhamtani and Berg-Kirkpatrick, 2018; Tu et al., 2021). Different from single-image captioning (Kim et al., 2019; Jiang et al., 2019; Fisch et al., 2020), change captioning addresses two-image caption- ing, which requires not only to understand both * This work was done at VIPL research group, CAS. † Corresponding author 1The code of this paper has been made publicly available at https://github.com/tuyunbin/R3Net. The tiny red metal block that is to the right of the matte cylinder moved. <Before> <After> <Change Captions> The black car has moved slightly down the road. Figure 1: Two examples of change captioning about an object move. The first example shows that the view- point changes the scale and location of the objects in the “after” image; the second example shows mostly well-aligned a pair of images with underlying illumina- tion changes from surveillance cameras. image content, but also to describe their disagree- ment. As the pioneer work, Jhamtani et al. (Jham- tani and Berg-Kirkpatrick, 2018) described seman- tic changes between mostly well-aligned image pairs with underlying illumination changes from surveillance cameras. However, they did not con- sider viewpoint changes that often happen in a dy- namic world, and image pairs cannot be mostly well aligned in this case. Hence, feature shift be- tween two unaligned images …

Content truncated to 500 words