Generating Visual Spatial Description via Holistic 3D Scene Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7960–7977 July 9-14, 2023 ©2023 Association for Computational Linguistics Generating Visual Spatial Description via Holistic 3D Scene Understanding Yu Zhao1,2, Hao Fei2∗, Wei Ji2, Jianguo Wei1 Meishan Zhang3, Min Zhang3, Tat-Seng Chua2 1 College of Intelligence and Computing, Tianjin University, China 2 Sea-NExT Joint Lab, National University of Singapore, Singapore 3 Harbin Institute of Technology (Shenzhen), China {zhaoyucs,jianguowei}@tju.edu.cn, {haofei37,jiwei,dcscts}@nus.edu.sg mason.zms@gmail.com, zhangmin2021@hit.edu.cn Abstract Visual spatial description (VSD) aims to gen- erate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical vision features, thus inevitably falling prey to the problem of skewed spatial understanding of target objects. In this work, we investigate the incorporation of 3D scene features for VSD. With an external 3D scene extractor, we ob- tain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (GO3D- S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes. Besides, we propose a scene subgraph select- ing mechanism, sampling topologically-diverse subgraphs from GO3D-S2G, where the diverse local structure features are navigated to yield spatially-diversified text generation. Experi- mental results on two VSD datasets demon- strate that our framework outperforms the base- lines significantly, especially improving on the cases with complex visual spatial relations. Meanwhile, our method can produce more spatially-diversified generation. Code is avail- able at https://github.com/zhaoyucs/VSD. 1 Introduction Visual spatial description is a newly emerged vision-language task, which aims to generate a tex- tual descriptive sentence of the spatial relationship between two target visual objects in a given image (Zhao et al., 2022). VSD falls into the category of image-to-text generation, while in particular focus- ing on the visual spatial semantics understanding, which has great values on the real-world human- computer interaction (HCI) applications (Heuser et al., 2020), e.g., automatic navigation (Pendão and Moreira, 2021; Wang et al., 2023b), personal assistance (Vanhooydonck et al., 2010), and un- manned manipulation (Castaman et al., 2021; Wang et al., 2023a, 2022a). ∗Corresponding author: Hao Fei ➢ 2D Space Modeling The gray car is parked on the left of the blue car. ➢ 3D Space Modeling The gray car is parked in front of the blue car. ✘ ✔ (a) Modeling 3D scene features results in correct spatial understanding The gray car is parked in front of the blue car. The gray car is parked in front of the blue car next to the building. The gray car near the house is parked in front of the blue car. The gray car near the house is parked in front of the blue car on the road. (b) Holistic 3D scene features help generate diversified spatial descriptions Figure 1: Examples of the visual spatial description. Zhao et al. (2022) pioneer the VSD task by manually annotating the spatial descriptions to the images based on the visual spatial classification datasets (Krishna …

Content truncated to 500 words