Scene Graph Parsing as Dependency Parsing

Proceedings of NAACL-HLT 2018, pages 397–407 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Scene Graph Parsing as Dependency Parsing Yu-Siang Wang National Taiwan University b03202047@ntu.edu.tw Chenxi Liu(�) Johns Hopkins University cxliu@jhu.edu Xiaohui Zeng Hong Kong University of Science and Technology xzengaf@connect.ust.hk Alan Yuille Johns Hopkins University alan.yuille@jhu.edu Abstract In this paper, we study the problem of parsing structured knowledge graphs from textual de- scriptions. In particular, we consider the scene graph representation (Johnson et al., 2015) that considers objects together with their at- tributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by in- troducing an alternative but equivalent edge- centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we com- bine the two-stage pipeline used in prior work (generic dependency parsing followed by sim- ple post-processing) into one, enabling end-to- end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.1 1 Introduction Recent years have witnessed the rise of interest in many tasks at the intersection of computer vision and natural language processing, including seman- tic image retrieval (Johnson et al., 2015; Vendrov et al., 2015), image captioning (Mao et al., 2014; Karpathy and Li, 2015; Donahue et al., 2015; Liu et al., 2017b), visual question answering (Antol et al., 2015; Zhu et al., 2016; Andreas et al., 2016), and referring expressions (Hu et al., 2016; Mao et al., 2016; Liu et al., 2017a). The pursuit for these tasks is in line with people’s desire for high level understanding of visual content, in particu- lar, using textual descriptions or questions to help understand or express images and scenes. 1Code is available at https://github.com/ Yusics/bist-parser/tree/sgparser What is shared among all these tasks is the need for a common representation to establish connec- tion between the two different modalities. The ma- jority of recent works handle the vision side with convolutional neural networks, and the language side with recurrent neural networks (Hochreiter and Schmidhuber, 1997; Cho et al., 2014) or word embeddings (Mikolov et al., 2013; Pennington et al., 2014). In either case, neural networks map original sources into a semantically meaningful (Donahue et al., 2014; Mikolov et al., 2013) vector representation that can be aligned through end-to- end training (Frome et al., 2013). This suggests that the vector embedding space is an appropriate choice as the common representation connecting different modalities (see e.g. Kaiser et al. (2017)). While the dense vector representation yields impressive performance, it has an unfortunate lim- itation of being less intuitive and hard to interpret. Scene graphs (Johnson et al., 2015), on the other hand, proposed a type of directed graph to encode information in terms of objects, attributes of ob- jects, and relationships …

Content truncated to 500 words