Question Modifiers in Visual Question Answering

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1472–1479 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1472 Question Modifiers in Visual Question Answering William Britton, Somdeb Sarkhel, Deepak Venugopal University of Memphis, Adobe Research, University of Memphis wjbrittonv@gmail.com, sarkhel@adobe.com, dvngopal@memphis.edu Abstract Visual Question Answering (VQA) is a challenge problem that can advance AI by integrating several important sub-disciplines including natural language understanding and computer vision. Large VQA datasets that are publicly available for training and evaluation have driven the growth of VQA models that have obtained increasingly larger accuracy scores. However, it is also important to understand how much a model understands the details that are provided in a question. For example, studies in psychology have shown that syntactic complexity places a larger cognitive load on humans. Analogously, we want to understand if models have the perceptual capability to handle modifications to questions. Therefore, we develop a new dataset using Amazon Mechanical Turk where we asked workers to add modifiers to questions based on object properties and spatial relationships. We evaluate this data on LXMERT which is a state-of-the-art model in VQA that focuses more extensively on language processing. Our conclusions indicate that there is a significant negative impact on the performance of the model when the questions are modified to include more detailed information. Keywords: visual question answering, modifiers, deep models, perception 1. Introduction Visual Question Answering (VQA) is a highly chal- lenging multi-disciplinary task that requires integration of several key disciplines including natural language understanding, computer vision and knowledge repre- sentation. Specifically, in VQA, a system is required to answer questions (that may be open-ended) that are based on a specific image. Due to its multi-disciplinary nature, it can be argued that advances made in this task can potentially be a significant step forward for AI in general. Based on the VQA benchmark dataset (Antol et al., 2015), several new models for VQA have been devel- oped over the past few years (Goyal et al., 2017; Sel- varaju et al., 2020; Tan and Bansal, 2019). While the goal is to enhance both visual and language understand- ing, it was observed in some cases that due to the type of questions, some priors based on language can signif- icantly improve accuracy. For example, as pointed out in (Goyal et al., 2017), a simple prior such as assign- ing the answer “yes” a high probability for a question that begins with “do you see” can often achieve very high scores. However, the use of such priors do not re- flect true understanding and are unlikely to generalize to real-world settings. To address this, in (Goyal et al., 2017), a balanced VQA dataset was developed to evaluate systems. Here, a large dataset was collected using Amazon Mechanical Turk where, the same ques- tion has different answers for different images. Fur- ther, the images with complementary answers are se- lected such that they are similar to each other. Thus, a model …

Content truncated to 500 words