CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3692–3709 June 6–11, 2021. ©2021 Association for Computational Linguistics 3692 CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images Shailaja Keyur Sampat∗, Akshay Kumar, Yezhou Yang and Chitta Baral Arizona State Universiy, USA {ssampa17,akuma216,yz.yang, chitta}@asu.edu Abstract Most existing research on visual question an- swering (VQA) is limited to information ex- plicitly present in an image or a video. In this paper, we take visual understanding to a higher level where systems are challenged to answer questions that involve mentally simulating the hypothetical consequences of performing spe- cific actions in a given scenario. Towards that end, we formulate a vision-language question answering task based on the CLEVR (Johnson et al., 2017a) dataset. Wethen modify the best existing VQA methods and propose baseline solvers for this task. Finally, we motivate the development of better vision-language models by providing insights about the capability of di- verse architectures to perform joint reasoning over image-text modality1. 1 Introduction In 2014, Michael Jordan, in an interview (Gomes, 2014) said that “Deep learning is good at certain problems like image classification and identifying objects in the scene, but it struggles to talk about how those objects relate to each other, or how a person/robot would interact with those objects. For example, humans can deal with inferences about the scene: what if I sit down on that?, what if I put something on top of something? etc. There exists a range of problems that are far beyond the capability of today’s machines." While this interview was six years ago, and since then there has been a lot of progress in deep learn- ing and its applications to visual understanding. Additionally, a large body of visual question an- swering (VQA) datasets (Antol et al., 2015; Ren et al., 2015; Hudson and Manning, 2019) have been compiled and many models have been developed ∗corresponding author 1Dataset setup scripts and code for baselines are made available at https://github.com/shailaja183/clevr_hyp. For ad- ditional details about the dataset creation process, refer sup- plementary material. Figure 1: Motivation for the proposed CLEVR_HYP dataset: an example demonstrating how humans can do mental simulations and reason over resulting scenario. over them, but the above mentioned “inferences about the scene” issue stated by Jordan remains largely unaddressed. In most existing VQA datasets, scene under- standing is holistic and questions are centered around information explicitly present in the image (i.e. objects, attributes and actions). As a result, ad- vanced object detection and scene graph techniques have been quite successful in achieving good per- formance over these datasets. However, provided an image, humans can speculate a wide range of implicit information. For example, the purpose of various objects in a scene, speculation about events that might have happened before, consider numer- ous imaginary situations and predicting possible future outcomes, intentions of a subject to perform particular actions, and many more. Among the above, an …

Content truncated to 500 words