Aligned Dual Channel Graph Convolutional Network for Visual Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7166–7176 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7166 Aligned Dual Channel Graph Convolutional Network for Visual Question Answering Qingbao Huang1,2, Jielong Wei2, Yi Cai1∗ Changmeng Zheng1, Junying Chen1, Ho-fung Leung3, Qing Li4 1School of Software Engineering, South China University of Technology, Guangzhou, China 2School of Electrical Engineering, Guangxi University, Nanning, Guangxi, China 3The Chinese University of Hong Kong, Hong Kong SAR, China 4The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China qbhuang@gxu.edu.cn, 1712306010@st.gxu.edu.cn, ycai@scut.edu.cn Abstract Visual question answering aims to answer the natural language question about a given im- age. Existing graph-based methods only focus on the relations between objects in an image and neglect the importance of the syntactic de- pendency relations between words in a ques- tion. To simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a ques- tion, we propose a novel dual channel graph convolutional network (DC-GCN) for better combining visual and textual advantages. The DC-GCN model consists of three parts: an I-GCN module to capture the relations be- tween objects in an image, a Q-GCN module to capture the syntactic dependency relation- s between words in a question, and an atten- tion alignment module to align image represen- tations and question representations. Experi- mental results show that our model achieves comparable performance with the state-of-the- art approaches. 1 Introduction As a form of visual Turing test, visual question answering (VQA) has drawn much attention. The goal of VQA (Antol et al., 2015; Goyal et al., 2017) is to answer a natural language question related to the contents of a given image. Attention mecha- nisms are served as the backbone of the previous mainstream approaches (Lu et al., 2016; Yang et al., 2016; Yu et al., 2017), however, they tend to catch only the most discriminative information, ignoring other rich complementary clues (Liu et al., 2019). Recent VQA studies have been exploring higher level semantic representation of images, notably using graph-based structures for better image under- standing, such as scene graph generation (Xu et al., 2017; Yang et al., 2018), visual relationship detec- tion (Yao et al., 2018), object counting (Zhang et al., ∗Corresponding author: Yi Cai (ycai@scut.edu.cn). Figure 1: (a) The question and the ground true answer. (b) The wrong answer is predicted by a state-of-the-art model, which focuses on the highlighted region in the image. The depth of the color indicates the weights of the words in the question, where deeper color repre- sents higher weight. The question is performed by syn- tactic dependency parsing. (c) The dependency parsing of the question is obtained by the universal Standford Dependencies tool (De Marneffe et al., 2014). 2018a), and relation reasoning (Cao et al., 2018; Li et al., 2019; Cadene et al., 2019a). Representing images as graphs allows one to explicitly model interactions between two objects in an image, so as to seamlessly transfer information between graph nodes …

Content truncated to 500 words