Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9212 - 9224 December 7-11, 2022 ©2022 Association for Computational Linguistics Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense Zhecan Wang1, Haoxuan You1, Yicheng He1, Wenhao Li1, Kai-Wei Chang2, Shih-Fu Chang1 1 Columbia University, New York, 2 University of California, Los Angeles {hy2612, rs4110, zw2627, sc250}@columbia.edu,kwchang@cs.ucla.edu Abstract Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross- reference in-between to fully integrate and achieve comprehension of the visual scene de- scribed. Recently, various approaches have been developed and have achieved high per- formance on visual commonsense benchmarks. However, it is unclear whether the models re- ally understand the visual scene and underly- ing commonsense knowledge due to limited evaluation data resources. To provide an in- depth analysis, we present a Multimodal Eval- uation (ME) pipeline to automatically gener- ate question-answer pairs to test models’ un- derstanding of the visual scene, text, and re- lated knowledge. We then take a step further to show that training with the ME data boosts model’s performance in standard VCR evalu- ation. Lastly, our in-depth analysis and com- parison reveal interesting findings: (1) semanti- cally low-level information can assist learning of high-level information but not the opposite; (2) visual information is generally under uti- lization compared with text. 1 Introduction Vision Language (VL) understanding is challeng- ing because it requires VL models to identify and integrate information from both modalities to fully understand visual scenes. Numerous VL benchmarks have been created such as CLEVER (Johnson et al., 2017), GQA (Hudson and Man- ning, 2019), VQA (Li et al., 2018), VCR (Zellers et al., 2019) and SNLI-VE (Xie et al., 2018). These benchmarks typically form VL evaluation in question-answering format with images and test models’ understanding of both VL modalities. De- spite the high accuracy achieved by existing large pretrained VL models, recent works have pointed out that VL models tend to exploit data biases such Figure 1: An example from VL benchmark, Visual Com- monsense Reasoning (VCR) (Zellers et al., 2019). VL models can answer the highly semantic VCR question correctly but fail terribly in answering related visual question (Q2), textual question (Q3), and background knowledge question (Q4). as shallow mappings with language priors and un- balanced utilization of information between modal- ities (Cao et al., 2020; Jimenez et al., 2022; Sel- varaju et al., 2020). As in Fig. 1, notwithstanding the model’s suc- cess in answering Q1, the same model fails to an- swer the related visual, textual, and background questions. This example demonstrates that the VL model does not fully understand the visual scene, which leads to prediction inconsistency (when one model makes conflicting(inconsistent) predictions in two related questions). In our analysis, predic- tion inconsistency is surprisingly common among models in different modalities. Former works have also pointed out that most VQA systems achieve only middling self-consistency (60 − 80%) 9212 (Jimenez et al., 2022). Therefore, we pose doubts to existing VL …

Content truncated to 500 words