Are Visual-Linguistic Models Commonsense Knowledge Bases?

Proceedings of the 29th International Conference on Computational Linguistics, pages 5542–5559 October 12–17, 2022. 5542 Are Visual-Linguistic Models Commonsense Knowledge Bases? Hsiu-Yu Yang University of Stuttgart hsiu-yu.yang@ims.uni-stuttgart.de Carina Silberer University of Stuttgart carina.silberer@ims.uni-stuttgart.de Abstract Despite the recent success of pretrained lan- guage models as on-the-fly knowledge sources for various downstream tasks, they have been shown to inadequately represent trivial com- mon facts that vision typically captures. This limits their application to natural language un- derstanding tasks that require commonsense knowledge. We seek to determine the capa- bility of pretrained visual–linguistic models as knowledge sources on demand. To this end, we systematically compare language-only and visual–linguistic models in a zero-shot com- monsense question answering inference task. We find that visual–linguistic models are highly promising regarding their benefit for text-only tasks on certain types of commonsense knowl- edge associated with the visual world. Sur- prisingly, this knowledge can be activated even when no visual input is given during inference, suggesting effective multimodal fusion during pretraining. However, we also reveal that there is still a huge scope for improvements towards better cross-modal reasoning abilities and pre- training strategies for event understanding.1 1 Introduction Commonsense knowledge is essential in human life for task solving and communication. Being aggregated knowledge acquired from past experi- ences and communications, it is usually left im- plicit in human communication, and used sub- consciously for reasoning and drawing inferences. This puts a challenge on natural language under- standing systems, and a large body of work has put forward approaches to provide commonsense knowledge to models for downstream tasks (Yang et al., 2019; Chen et al., 2018; Mihaylov and Frank, 2018). While knowledge bases have been a pop- ular way to provide relevant knowledge for a task 1Our datasets, CWWVImg and CWWVClip, are provided at https://github.com/Mallory24/CS_Probing at hand, more recently, pretrained language mod- els (PTLMs) have become a popular mechanism to extract knowledge in free-form text on demand. Tasks range from, e.g., persona-grounded dialog (Majumder et al., 2020), narrative story generation (Ammanabrolu et al., 2020), to metaphor genera- tion (Stowe et al., 2021). Shwartz and Choi (2020) and Bisk et al. (2020), however, suggest that text corpora alone may be insufficient for knowledge ac- quisition due to reporting bias found in them (Gor- don and Durme, 2013). This has led to analyzing the knowledge that PTLMs possess through dedi- cated probing studies (Petroni et al., 2019; Singh et al., 2021; Zhou et al., 2020b) Existing works on probing PTLMs have used a loose categorization of commonsense, which lim- its a comprehensive understanding of the types of commonsense they possess and lack, respectively. At the same time, the literature proposes vision as a promising knowledge source (Izadinia et al., 2015; Bagherinezhad et al., 2016; Sadeghi et al., 2015). It seems therefore straightforward to leverage visual– linguistic (VL) models for knowledge extraction on demand—these representation models are ex- tensions of PTLMs to the visual–linguistic domain by pretraining LMs and image recognition models jointly on multimodal data (Tan and Bansal, 2019; Chen et …

Content truncated to 500 words