‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2766–2771 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2766 ‘Just because you are right, doesn’t mean I am wrong’: Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, Chitta Baral Arizona State University, Tempe, AZ, USA {mluo26,ssampa17,rtallman,yzeng55,mvancha,asajja,chitta}@asu.edu Abstract GQA (Hudson and Manning, 2019) is a dataset for real-world visual reasoning and compo- sitional question answering. We found that many answers predicted by the best vision- language models on the GQA dataset do not match the ground-truth answer but still are se- mantically meaningful and correct in the given context. In fact, this is the case with most exist- ing visual question answering (VQA) datasets where they assume only one ground-truth an- swer for each question. We propose Alter- native Answer Sets (AAS) of ground-truth answers to address this limitation, which is created automatically using off-the-shelf NLP tools. We introduce a semantic metric based on AAS and modify top VQA solvers to sup- port multiple plausible answers for a ques- tion. We implement this approach on the GQA dataset and show the performance improve- ments. 1 Introduction One important style of visual question answering (VQA) task involves open-ended responses such as free-form answers or fill-in-the-blanks. The possi- bility of multiple correct answers and multi-word responses makes the evaluation of open-ended tasks harder, which has forced VQA datasets to restrict answers to be a single word or a short phrase. Despite enforcing these constraints, from our analysis of the GQA dataset (Hudson and Man- ning, 2019), we noticed that a significant portion of the visual questions have issues. For example, a question “Who is holding the bat?” has only one ground truth answer “batter” while other rea- sonable answers like “batsman”, “hitter” are not credited. We identified six different types of issues with the dataset and illustrated them in Table 1. A large-scale human-study conducted by (Gurari and Grauman, 2017) on VQA (Antol et al., 2015) and VizWiz (Gurari et al., 2019) found that almost 50% questions in these datasets have multiple pos- sible answers. datasets had similar observations. The above evidence suggests that it is unfair to pe- nalize models if their predicted answer is correct in a given context but does not match the ground truth answer. With this motivation, we leverage existing knowledge bases and word embeddings to generate Alternative Answer Sets (AAS) instead of consider- ing visual questions to have fixed responses. Since initially obtained AAS are generated from multiple sources and observed to be noisy, we use textual entailment to verify semantic viability of plausi- ble answers to make alternative answer sets more robust. We justify the correctness and quality of the generated AAS by human evaluation. We in- troduce a semantic metric based on AAS and train two vision-language models LXMERT (Tan and Bansal, 2019) and ViLBERT (Lu …

Content truncated to 500 words