LRRA:A Transparent Neural-Symbolic Reasoning Framework for Real-World Visual Question Answering

LRRA:A Transparent Neural-Symbolic Reasoning Framework for Real-World Visual Question Answering Zhang Wan∗, Keming Chen, Yujie Zhang†, Jinan Xu, Yufeng Chen School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China {19120413,20120341,yjzhang,jaxu,chenyf}@bjtu.edu.cn Abstract The predominant approach of visual question answering (VQA) relies on encoding the image and question with a ”black box” neural encoder and decoding a single token into answers such as ”yes” or ”no”. Despite this approach’s strong quantitative results, it struggles to come up with human-readable forms of justification for the prediction process. To address this insufficiency, we propose LRRA[Look,Read,Reasoning,Answer],a transparent neural-symbolic framework for visual question answering that solves the complicated problem in the real world step-by-step like humans and provides human-readable form of justification at each step.Specifically, LRRA learns to first convert an image into a scene graph and parse a question into multiple reasoning instructions. It then executes the reasoning instructions one at a time by traversing the scene graph using a recurrent neural-symbolic execution module.Finally, it generates answers to the given questions and makes corresponding marks on the image. Furthermore,we believe that the relations between objects in the question is of great significance for obtaining the correct answer, so we create a perturbed GQA test set by removing linguistic cues (attributes and relations) in the questions to analyze which part of the question contributes more to the answer.Our experiments on the GQA dataset show that LRRA is significantly better than the existing representative model (57.12% vs. 56.39%). Our experiments on the perturbed GQA test set show that the relations between objects is more important for answering complicated questions than the attributes of objects. Keywords:Visual Question Answering, Relations Between Objects, Neural-Symbolic Reason- ing. 1 Introduction Currently, the predominant approach to visual question answering (VQA)relies on encoding the image and question with a black-box transformer encoder (H.Tan et al., 2019; J.Lu et al., 2020). These works carry out complex calculations behind the scenes but only produce a single token as prediction output (for example, ”yes”, ”no”) and they can not provide an easy-to-understand form of justification consistent with their predictions. In addition, recent studies have shown that the end-to-end model can be easily optimized to learn the ”shortcut bias” of the data set instead of reasoning (for example, the model uses the implicit fused question representations (S.Antol et al., 2015; Y. Goyal et al., 2017), the answer can be directly inferred according to certain language patterns), which tend to undesirably adhere to superficial or even potentially misleading statistical associations(A.Agrawal et al., 2016), so they do not really understand the question, and often perform poorly in the face of complex reasoning problems in the real world. In order to solve the above insuficiencys, we learn the correct problem solving process step-by-step mimicking humans and propose a neural-symbolic approach for visual question answering that fully disentangles vision and language understanding from reasoning. A human would first (1) ∗First author †Corresponding author ©2021 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 International License CCL …

Content truncated to 500 words