SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

Findings of the Association for Computational Linguistics: EACL 2023, pages 149–157 May 2-6, 2023 ©2023 Association for Computational Linguistics FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering Weizhe Lin Department of Engineering University of Cambridge United Kingdom wl356@cam.ac.uk Zhilin Wang Department of Linguistics University of Washington United States zhilinw@uw.edu Bill Byrne Department of Engineering University of Cambridge United Kingdom bill.byrne@eng.cam.ac.uk Abstract The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually- grounded questions that require information re- trieval using common sense knowledge graphs to answer. It has been observed that the original dataset is highly imbalanced and concentrated on a small portion of its associated knowledge graph. We introduce FVQA 2.0 which con- tains adversarial variants of test questions to address this imbalance. We show that systems trained with the original FVQA train sets can be vulnerable to adversarial samples and we demonstrate an augmentation scheme to reduce this vulnerability without human annotations. 1 Introduction Knowledge-based Visual Question Answering (KB- VQA) lies at the intersection of Computer Vision, Natural Language Processing, and Information Re- trieval. A KB-VQA system must access external knowledge sources to find a correct and complete answer, a task that is sometimes hard for humans. Fact-based Visual Question Answering (FVQA) (Wang et al., 2017) is a VQA task in which visually-grounded questions and answers about images are grounded by knowledge-graph (KG) triplets taken from several ‘common sense’ knowledge bases, such as ConceptNet (Speer et al., 2017), Webchild (Tandon et al., 2017), and DBpedia (Auer et al., 2007). For instance, “Question: Which thing in the image can be used for scooping food? Answer: spoon” is associated with the KG triplet “spoon - UsedFor - scooping food”. These questions are challenging in that retrieving information from external KGs is necessary. The original FVQA dataset (Wang et al., 2017) has several readily observed limitations. First, the dataset is small (5486 samples) and the annotations are limited to a single answer per question, ignor- ing other correct answers. This limitation arises from the FVQA creation process in which annota- tors were first asked to select a KG triplet on which they would ask a question about an image. This approach prevented the annotators from labeling other valid KG triplets. Secondly, the dataset is highly imbalanced. Some triplets and answers are frequently used, but other KG triplets and answers are severely underrepresented in training. For ex- ample, there are 1,129 possible answers in total, but over 90% of questions focus on only a half of them; 792 (70%) answers appear less than 3 times; only 4,216 out of ∼220k triplets are used. These limitations lead to a potential problem: KB-VQA systems trained on this dataset overfit on these frequently used triplets and perform poorly on variants that contain other valid triplets or other images. Also, extensive overlap between training and test can lead to an unrealistically high ques- tion answering baseline performance. We noted that a question with a triplet unseen in training is often answered with ‘person’, since …

Content truncated to 500 words