EaSe: A Diagnostic Tool for VQA based on Answer Diversity

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2407–2414 June 6–11, 2021. ©2021 Association for Computational Linguistics 2407 EaSe: A Diagnostic Tool for VQA Based on Answer Diversity Shailza Jolly TU Kaiserslautern, Germany DFKI GmbH, Germany shailza.jolly@dfki.de Sandro Pezzelle ILLC, University of Amsterdam, Netherlands s.pezzelle@uva.nl Moin Nabi SAP AI Research, Germany m.nabi@sap.com Abstract We propose EASE, a simple diagnostic tool for Visual Question Answering (VQA) which quantifies the difficulty of an image, question sample. EASE is based on the pattern of an- swers provided by multiple annotators to a given question. In particular, it considers two aspects of the answers: (i) their Entropy; (ii) their Semantic content. First, we prove the validity of our diagnostic to identify samples that are easy/hard for state-of-art VQA models. Second, we show that EASE can be success- fully used to select the most-informative sam- ples for training/fine-tuning. Crucially, only information that is readily available in any VQA dataset is used to compute its scores.1 1 Introduction Visual Question Answering (VQA; Antol et al., 2015) requires models to jointly understand an image and a natural language question. This is a challenging task; despite massive training data and recent pre-training strategies (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2020) models still struggle to close the gap with oracle performance. VQA datasets (e.g., Goyal et al., 2017; Gurari et al., 2018) consist of ⟨image, question⟩ pairs for which N human annotators have provided an an- swer in natural language. When trained on these samples, VQA models are fed with the most fre- quently chosen answer in the pattern. During in- ference, the answer with the highest probability is evaluated against the pattern of N ground-truth an- swers. According to the standard VQA metric (An- tol et al., 2015), a model’s prediction is considered as perfectly correct if it matches an answer that was frequent in the pattern; less accurate if matching an underrepresented one. This metric implies that, for the majority of cases, several annotators agree on the same exact answer—and a model can thus achieve 100% accuracy in the task. On the other 1Code at: github.com/shailzajolly/EaSe Q: What is the pattern of the little girl's dress? GT: plaid: 4, checks and flowers: 1, checkered with flowers: 1, polka dots, squares, plaid: 1, squares and flowers: 1, flowers: 1, plaid and floral: 1 EaSe: 1.0 Q: Where is this? GT: road: 4, outside: 2, pakistan: 1, outdoors: 1, sidewalk: 1, sweden: 1 EaSe: 0.30 Figure 1: One image from VQA2.0 with two questions and the answers by 10 annotators. Frequency of each unique answer (e.g., plaid : 4) and EASE values of the samples (the higher, the easier) are reported. hand, this suggests that various ⟨image, question⟩ pairs can have different patterns of answers; i.e., they can be more or less scattered depending on the features of the question, the image, or both. In Fig. 1, the annotators did not converge …

Content truncated to 500 words