Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1437 - 1449 December 7-11, 2022 ©2022 Association for Computational Linguistics Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation Yu Zhao1, Jianguo Wei1, Zhichao Lin2, Yueheng Sun1, Meishan Zhang3∗, Min Zhang3 1College of Intelligence and Computing, Tianjin University 2School of New Media and Communication, Tianjin University 3Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) {zhaoyucs,jianguo,chaosmyth,yhs}@tju.edu.cn, {zhangmeishan,zhangmin2021}@hit.edu.cn Abstract Image-to-text tasks, such as open-ended image captioning and controllable image description, have received extensive attention for decades. Here, we further advance this line of work by presenting Visual Spatial Description (VSD), a new perspective for image-to-text toward spa- tial semantics. Given an image and two objects inside it, VSD aims to produce one description focusing on the spatial perspective between the two objects. Accordingly, we manually anno- tate a dataset to facilitate the investigation of the newly-introduced task and build several bench- mark encoder-decoder models by using VL- BART and VL-T5 as backbones. In addition, we investigate pipeline and joint end-to-end architectures for incorporating visual spatial re- lationship classification (VSRC) information into our model. Finally, we conduct experi- ments on our benchmark dataset to evaluate all our models. Results show that our models are impressive, providing accurate and human-like spatial-oriented text descriptions. Meanwhile, VSRC has great potential for VSD, and the joint end-to-end architecture is the better choice for their integration. We make the dataset and codes public for research purposes.1 1 Introduction Text generation from images is a widely-adopted means for deep understanding of cross-modal data that has received increasing interest of both com- puter vision (CV) and natural language processing (NLP) communities (He and Deng, 2017). Image- to-text tasks generate natural language texts to as- sist in understanding the scene meaning of a spe- cific image, which might be beneficial for a variety of applications such as image retrieval (Diao et al., 2021; Ahmed et al., 2021), perception assistance (Xu et al., 2018; Shashirangana et al., 2021), pedes- trian detection (Hasan et al., 2021), and medical system (Miura et al., 2021). ∗Corresponding author 1https://github.com/zhaoyucs/VSD Task Condition Target Text Image Captioning —— A man is walking past a car. VSR-guided Captioning walk; ⟨Arg⟩, ⟨Loc⟩ A man is walking cross a street. Visual Question Answering What color is the car? The car is red. Our Task: VSD ⟨man, car⟩ A man is walking behind a red car from right to left. ⟨car, pole⟩ A red car is parked to the left of a pole. Figure 1: A comparison of three example image-to-text generation tasks and the proposed VSD in this work. Image-to-text tasks take on various forms when serving different purposes. Figure 1 illustrates a comparison of three example tasks. First, the generic open-ended image captioning aims to pro- vide a summarised description that describes an in- put image and reflects the overall understanding of the image (Lindh et al., 2020; Vinyals et al., 2015; Ji et al., 2020). Furthermore, the verb-specific se- mantic roles (VSR) guided captioning (Chen …

Content truncated to 500 words