Visual Commonsense in Pretrained Unimodal and Multimodal Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5321 - 5335 July 10-15, 2022 ©2022 Association for Computational Linguistics Visual Commonsense in Pretrained Unimodal and Multimodal Models Chenyu Zhang Benjamin Van Durme Zhuowan Li∗ Elias Stengel-Eskin∗ Johns Hopkins University {czhan105, vandurme, zli110, elias}@jhu.edu Abstract Our commonsense knowledge about objects in- cludes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world- knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree uni- modal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we cre- ate the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data cor- relates much better than ungrounded text-only data with crowdsourced color judgments pro- vided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct at- tribute distributions, but are still subject to re- porting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.1 1 Introduction The observation that human language understand- ing happens in a rich multimodal environment has led to an increased focus on visual grounding in natural language processing (NLP) (Baltrusaitis et al., 2019; Bisk et al., 2020), driving comparisons between traditional unimodal text-only models and multimodal models which take both text and image inputs. In this work, we explore to what extent uni- modal and multimodal models are able to capture commonsense visual concepts across five types of relations: color, shape, material, size, and visual co- occurrence (cf. Fig. 1). We further explore how this *Joint Advising 1The dataset and code is available at https://github. com/ChenyuHeidiZhang/VL-commonsense. ability is influenced by reporting bias (Gordon and Van Durme, 2013), the tendency of large corpora to over- or under-report events. We define visual commonsense as knowledge about generic visual concepts, e.g. “knobs are usually round”, and we measure this knowledge via frequency distributions over potential properties (e.g. round, square, etc). A visually-informed language model should be able to capture such properties. Our color, shape, mate- rial, and co-occurrence data are mined from Visual Genome (Krishna et al., 2016), and our size data are created from object lists. They contain a large number of examples of per-object attribute distri- butions and “object-attribute” pairs. Paik et al. (2021) evaluate language models’ color perception using a human-annotated color dataset (CoDa), finding that reporting bias nega- tively influences model performance and that mul- timodal training can mitigate those effects. In this work, we confirm those findings while extending the evaluation to a broader range of visually salient properties, resulting in a more comprehensive met- ric for visual …

Content truncated to 500 words