Scene Graph Modification as Incremental Structure Expanding

Proceedings of the 29th International Conference on Computational Linguistics, pages 5707–5720 October 12–17, 2022. 5707 Scene Graph Modification as Incremental Structure Expanding Xuming Hu1∗, Zhijiang Guo2∗, Yu Fu1, Lijie Wen1†, Philip S. Yu1,3 1Tsinghua University 2University of Cambridge 3University of Illinois at Chicago 1{hxm19,fy20}@mails.tsinghua.edu.cn 2zg283@cam.ac.uk 1wenlj@tsinghua.edu.cn 3psyu@uic.edu Abstract A scene graph is a semantic representation that expresses the objects, attributes, and relation- ships between objects in a scene. Scene graphs play an important role in many cross modality tasks, as they are able to capture the interac- tions between images and texts. In this paper, we focus on scene graph modification (SGM), where the system is required to learn how to up- date an existing scene graph based on a natural language query. Unlike previous approaches that rebuilt the entire scene graph, we frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE). ISE constructs the target graph by incrementally expanding the source graph without changing the unmodified structure. Based on ISE, we further propose a model that iterates between nodes prediction and edges prediction, infer- ring more accurate and harmonious expansion decisions progressively. In addition, we con- struct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets. Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins. Source code and data are available1. 1 Introduction A scene graph is a structural representation that cap- tures the semantics of visual scenes by encoding object instances, attributes of objects, and relation- ships between objects. (Johnson et al., 2015). As shown in Figure 1, the scene graph encodes ob- jects (e.g. “Boy”, “Racket”), attributes (e.g. “Girl is standing”), and relations (“Boy holding racket”). Scene graphs are able to capture the interactions between text and images by associating objects in the graph with regions of an image and modeling the relations between objects. Therefore, it has 1https://github.com/THU-BPM/SGM ∗ Equally Contributed. † Corresponding Author. I would like to see a girl holding racket. Holding Racket Standing Blue Holding Girl Racket Standing Blue Boy Objects Relations Attributes Figure 1: Example images and their corresponding scene graphs. Given the query, the original scene graph (left) is modified to be the target scene graph (right). been used in the cross modality task such as im- age retrieval, image captioning, and visual question answering (Schuster et al., 2015; Shi et al., 2019; Yang et al., 2019; Wang et al., 2020b). Recently, modifying the scene graph based on the input becomes an emerging research direction as cross-modal systems may need to resort to an in- teractive process through multiple iterations (Ram- nath et al., 2019; He et al., 2020). Take text-based image retrieval as an example, users start with a query describing the main objects or topics they are looking for, then modify the query to add more constraints or provide additional information based on previous search results. Instead of directly ma- nipulating images, scene graphs can be used to …

Content truncated to 500 words