MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4032–4059 July 9-14, 2023 ©2023 Association for Computational Linguistics MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks Letitia Parcalabescu and Anette Frank Computational Linguistics Department Heidelberg University Abstract Vision and language models (VL) are known to exploit unrobust indicators in individual modal- ities (e.g., introduced by distributional biases) instead of focusing on relevant information in each modality. That a unimodal model achieves similar accuracy on a VL task to a multimodal one, indicates that so-called uni- modal collapse occurred. However, accuracy- based tests fail to detect e.g., when the model prediction is wrong, while the model used rele- vant information from a modality. Instead, we propose MM-SHAP, a performance-agnostic multimodality score based on Shapley values that reliably quantifies in which proportions a multimodal model uses individual modali- ties. We apply MM-SHAP in two ways: (1) to compare models for their average degree of multimodality, and (2) to measure for indi- vidual models the contribution of individual modalities for different tasks and datasets. Ex- periments with six VL models – LXMERT, CLIP and four ALBEF variants – on four VL tasks highlight that unimodal collapse can oc- cur to different degrees and in different direc- tions, contradicting the wide-spread assump- tion that unimodal collapse is one-sided. Based on our results, we recommend MM-SHAP for analysing multimodal tasks, to diagnose and guide progress towards multimodal integration. Code available at https://github.com/ Heidelberg-NLP/MM-SHAP. 1 Introduction Vision and language (VL) tasks are dominated by general-purpose pretrained transformer-based VL models (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Li et al., 2020, 2021a). But we are only starting to understand why these multimodal (MM) models work so well, and how they utilise and fuse image and text modalities (Hessel and Lee, 2020; Cao et al., 2020). Even worse, these highly parametrised neural VL mod- els, pretrained on large amounts of data, tend to Figure 1: We display image-sentence alignment scores (ISA) and the textual degree T-SHAP that measures how much models focus on text rather than the image (with 100−T-SHAP% the corresponding visual degree) for 3 VL models. Blue/red highlights on text tokens and image tokens (patches) contribute towards higher/lower ISA. Note: CLIP’s ISA is an absolute score, while AL- BEF and LXMERT predict ISA probabilities. See Sec- tion 4.4 for more details on this figure; App. C for more detailed analysis of this instance and more samples. exploit artefacts and statistical correlations in the data (Shekhar et al., 2019; Kafle et al., 2019), show- ing little to no evidence of detailed linguistic or visual understanding (Milewski et al., 2022; Parcal- abescu et al., 2022; Thrush et al., 2022). Statistical biases towards indicators in one modality – to the 4032 detriment of others – can cause unimodal collapse (Parcalabescu et al., 2022), where seemingly MM models exploit one modality that exhibits biases, meaning that …

Content truncated to 500 words