Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9847–9857 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9847 Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers Stella Frank∗,D Emanuele Bugliarello∗,C Desmond ElliottC DUniversity of Trento CUniversity of Copenhagen stella.frank@unitn.it {emanuele, de}@di.ku.dk Abstract Pretrained vision-and-language BERTs aim to learn representations that combine informa- tion from both modalities. We propose a di- agnostic method based on cross-modal input ablation to assess the extent to which these models actually integrate cross-modal infor- mation. This method involves ablating inputs from one modality, either entirely or selec- tively based on cross-modal grounding align- ments, and evaluating the model prediction performance on the other modality. Model performance is measured by modality-specific tasks that mirror the model pretraining ob- jectives (e.g. masked language modelling for text). Models that have learned to construct cross-modal representations using both modal- ities are expected to perform worse when in- puts are missing from a modality. We find that recently proposed models have much greater relative difficulty predicting text when visual information is ablated, compared to predicting visual object categories when text is ablated, indicating that these models are not symmetri- cally cross-modal. 1 Introduction Vision-and-language (V&L) BERT models extend the BERT architecture (Devlin et al., 2019) to pro- duce cross-modal contextualised representations of multimodal inputs (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2020; Su et al., 2020). These models have proven to be highly effective when fine-tuned for a range of downstream tasks. How- ever, in spite of their versatility, little is known about how these models use cross-modal infor- mation: do their learned representations for lan- guage tasks include visual information (vision-for- language) and vice-versa (language-for-vision)? Su et al. (2020) claim that their model will use visual context for predicting masked words, thus aligning ∗Equal contribution. [MASK] playing tennis [MASK] playing tennis [MASK] playing tennis [MASK] playing tennis [MASK][MASK][MASK] Girl playing tennis None Object All None Phrase All Vision-for-Language Diagnostic Language-for-Vision Diagnostic p([MASK]=girl) p( = ) Figure 1: Cross-modal input ablation tests how well models predict masked data, given ablated inputs in the other modality. The Vision-for-Language Diagnos- tic (top) measures the effect of ablation, of either the aligned object or the full image, on masked token pre- diction, while the Language-for-Vision Diagnostic (bot- tom) measures the effect of ablating either the aligned Phrase or the entire sentence when predicting the prop- erties of a masked image region. visual and linguistic contexts, while Lu et al. (2020) hope to force their model to rely more heavily on language to predict image content. The extent to which these statements hold is unknown, in part because of the difficulty of analysing exactly how these models use information across modalities. In this paper, we introduce a cross-modal input ablation method to quantify the degree to which a pretrained model has learned to use cross-modal information. This method, which requires no ad- ditional training, involves ablating all or some of the inputs from one …

Content truncated to 500 words