Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2115–2129 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2115 Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning Da Yin Liunian Harold Li Ziniu Hu Nanyun Peng Kai-Wei Chang Computer Science Department, University of California, Los Angeles {da.yin,liunian.harold.li,bull,violetpeng,kwchang}@cs.ucla.edu gd-vcr.github.io Abstract Commonsense is defined as the knowledge that is shared by everyone. However, certain types of commonsense knowledge are corre- lated with culture and geographic locations and they are only shared locally. For ex- ample, the scenarios of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models’ ability to understand cultural and geo-location-specific commonsense. In particular, we study two state-of-the-art Vision-and-Language models, VisualBERT and ViLBERT trained on VCR, a standard multimodal commonsense bench- mark with images primarily from Western re- gions. We then evaluate how well the trained models can generalize to answering the ques- tions in GD-VCR. We find that the perfor- mance of both models for non-Western regions including East Asia, South Asia, and Africa is significantly lower than that for Western region. We analyze the reasons behind the performance disparity and find that the per- formance gap is larger on QA pairs that: 1) are concerned with culture-related scenarios, e.g., weddings, religious activities, and festi- vals; 2) require high-level geo-diverse com- monsense reasoning rather than low-order per- ception and recognition. Dataset and code are released at https://github.com/ WadeYin9712/GD-VCR. 1 Introduction Commonsense reasoning endows machines with high-level reasoning ability to understand situa- tions with implicit commonsense knowledge. Sup- pose that there is a scene where a woman is wearing a bridal gown at a party. An ideal AI system with commonsense knowledge should be able to infer that this woman is attending a wedding and likely to be the bride. Recently, the field of commonsense reasoning is progressing with the development of large-scale benchmark datasets (Zellers et al., 2018; Talmor et al., 2019), intended to cover a wide range of commonsense knowledge, such as physical inter- actions (Bisk et al., 2020), social conventions (Sap et al., 2019), and commonsense grounded in vision (Zellers et al., 2019). However, existing benchmarks are often com- posed by data from sources in certain regions (e.g., Western movies) and overlook the differences across groups in different regions1 due to factors including cultural differences. In the aforemen- tioned wedding example, while brides are usually in white in Western weddings, they often wear red and their faces are covered with a red cloth in tra- ditional Chinese weddings. If a model is unaware of regional characteristics or incapable of captur- ing the nuanced regional characteristics, it leads to a disparity in performance across different re- gions (Acharya et al., 2020). This motivates us to study how well a model trained on existing com- monsense annotations can generalize to tasks re- quiring commonsense …

Content truncated to 500 words