Scene Graph Modification Based on Natural Language Commands

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 972–990 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 972 Scene Graph Modification Based on Natural Language Commands Xuanli He† Quan Hung Tran‡ Gholamreza Haffari† Walter Chang‡ Trung Bui‡ Zhe Lin‡ Franck Dernoncourt‡ Nhan Dam† † Monash University, Clayton, Australia {xuanli.he1, gholamreza.haffari, nhan.dam}@monash.edu ‡Adobe Research, San Jose, CA {qtran, wachang, bui, dernonco}@adobe.com Abstract Structured representations like graphs and parse trees play a crucial role in many Natu- ral Language Processing systems. In recent years, the advancements in multi-turn user in- terfaces necessitate the need for controlling and updating these structured representations given new sources of information. Although there have been many efforts focusing on im- proving the performance of the parsers that map text to graphs or parse trees, very few have explored the problem of directly manip- ulating these representations. In this paper, we explore the novel problem of graph modifica- tion, where the systems need to learn how to update an existing scene graph given a new user’s command. Our novel models based on graph-based sparse transformer and cross at- tention information fusion outperform previ- ous systems adapted from the machine trans- lation and graph generation literature. We fur- ther contribute our large graph modification datasets to the research community to encour- age future research for this new problem. 1 Introduction Parsing text into structured semantics representa- tion is one of the most long-standing and active research problems in NLP. Numerous parsing meth- ods have been developed for many different seman- tic structure representations (Chen and Manning, 2014; Mrini et al., 2019; Zhou and Zhao, 2019; Clark et al., 2018; Wang et al., 2018). However, most of these previous works focus on parsing a single sentence, while a typical human-computer interaction session or conversation is not single- turn. A prominent example is image search. Users usually start with short phrases describing the main objects or topics they are looking for. Depend- ing on the result, the users may then modify their query to add more constraints or give additional information. In this case, without the modification capability, a static representation is not suitable to track the changing intent of the user. We argue that the back-and-forth and multi-turn nature of human-computer interactions necessitate the need for updating the structured representation. Another advantage of modifying a structured representation in the interactive setting is that it makes it easier to check the consistency. For instance, it is much easier to check whether the user requests two con- tradicting attributes for the same object in a scene graph during the interactive search, which can be done automatically. In this paper, we propose the problem of scene graph modification for search. A scene graph (John- son et al., 2015) is a semantic formalism which represents the desired image as a graph of objects with relations and attributes. This semantic repre- sentation has been shown to be very successful in retrieval systems (Johnson et al., 2015; Schuster et al., 2015; Vendrov …

Content truncated to 500 words