A Corpus for Visual Question Answering Annotated with Frame Semantic Information

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5524–5531 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5524 A Corpus for Visual Question Answering Annotated with Frame Semantic Information Mehrdad Alizadeh, Barbara Di Eugenio Department of Computer Science, University of Illinois at Chicago, Chicago, IL USA {maliza2, bdieugen}@uic.org Abstract Visual Question Answering (VQA) has been widely explored as a computer vision problem, however enhancing VQA systems with linguistic information is necessary for tackling the complexity of the task. The language understanding part can play a major role especially for questions asking about events or actions expressed via verbs. We hypothesize that if the question focuses on events described by verbs, then the model should be aware of or trained with verb semantics, as expressed via semantic role labels, argument types, and/or frame elements. Unfortunately, no VQA dataset exists that includes verb semantic information. We created a new VQA dataset annotated with verb semantic information called imSituVQA. imSituVQA is built by taking advantage of the imSitu dataset annotations. The imSitu dataset consists of images manually labeled with semantic frame elements, mostly taken from FrameNet. Keywords: Visual Question Answering, Verb Semantics, FrameNet, Data Augmentation 1. Introduction The goal of a Visual Question Answering (VQA) sys- tem is to answer user questions about an image (Antol et al., 2015b). Neural network based VQA models require large datasets in order to be trained efficiently (Kafle and Kanan, 2017). Currently available datasets approach the task mostly from a visual point of view. The questions are usually about objects, object attributes, object presence, object frequency, spatial reasoning and so on. However we believe the language component should play a major role as well. Exploring currently available VQA datasets, we realized a considerable portion of questions involve a verb other than ”to be”. We analyzed the VQA dataset (Antol et al., 2015a) since it has been widely used and also includes open-ended free-form questions. As shown in Figure 1, 43% of questions involve a verb other than ”to be in the VQA dataset. In order to compute the distribution of verbs, we used an automatic semantic role labeler. For example the output labels for the question: ”What is the bride wear- ing on her head?” is: V:wear.01 A0:bride AM-LOC:on R- A1:what. V:verb.0x pattern is used to check whether verb equals ’be’ or not. Therefore, we hypothesize that event verbs such as cook or catch, inherently provide semantic information that may help in answering questions about im- ages describing such events. Semantic information about verbs includes the type of arguments a verb can take and how the arguments partici- pate in the event expressed by a verb, but this information is missing in current VQA systems. A VQA system aware of or trained with such semantic information can narrow down possible responses. For example, the answer to the question ”What is the man cooking?”, should be constrained to be about food. However, neither do VQA datasets encode, nor …

Content truncated to 500 words