Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5444–5455 July 9-14, 2023 ©2023 Association for Computational Linguistics Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment Rohan Pandey Rulin Shao Paul Pu Liang Ruslan Salakhutdinov Louis-Philippe Morency Language Technologies Institute and Machine Learning Department Carnegie Mellon University {rspandey,rulins}@cs.cmu.edu Abstract Despite recent progress towards scaling up mul- timodal vision-language models, these mod- els are still known to struggle on compo- sitional generalization benchmarks such as Winoground. We find that a critical component lacking from current vision-language models is relation-level alignment: the ability to match di- rectional semantic relations in text (e.g., ‘mug in grass’) with spatial relationships in the im- age (e.g., the position of the mug relative to the grass). To tackle this problem, we show that relation alignment can be enforced by en- couraging the language attention from ‘mug’ to ‘grass’ (capturing the semantic relation ‘in’) to match the visual attention from the mug to the grass (capturing the corresponding physi- cal relation). Tokens and their corresponding objects are softly identified using a weighted mean of cross-modal attention. We prove that this notion of soft cross-modal equivalence is equivalent to enforcing congruence between vi- sion and language attention matrices under a ‘change of basis’ provided by the cross-modal attention matrix. Intuitively, our approach projects visual attention into the language at- tention space to calculate its divergence from the actual language attention, and vice versa. We apply our Cross-modal Attention Congru- ence Regularization (CACR) loss to fine-tune UNITER and improve its Winoground Group score by 5.75 points. 1 Introduction Compositionality is the ability to combine mean- ings of constituents according to structured rules. Recent work shows that Vision-Language Models (VLMs) fail to construct compositional represen- tations and generally ignore syntactic & structural information [Thrush et al., 2022, Milewski et al., 2022, Liang et al., 2022]. Winoground [Thrush et al., 2022] is a vision-language compositionality task that tests a VLM’s ability to match syntactic permutations of text with their visual interpreta- mug in grass grass in mug mug grass mug grass Global Alignment Entity Alignment Relation Alignment (Ours) in mug grass in mug grass in in Figure 1: Global Alignment (GA) only aligns the entire image with the corresponding caption. Entity Align- ment (EA) extracts entities from the image and caption for finer-grained alignment. Relation Alignment (RA) cross-modally aligns the intra-modal relations between entities in both the image and the text. We show RA is vital to improve compositional performance. tions, for example correctly matching “grass in mug” and “mug in grass” to their corresponding images. Winoground finds that all recent state- of-the-art VLMs perform below chance levels on this compositionality task. Contemporaneously, Milewski et al. [2022] probe for structural knowl- edge in VLMs, finding that they encode signifi- cantly less linguistic syntax than Language Models (LMs) and virtually no visual structure. Recently, Yuksekgonul et al. [2022] built a large dataset con- firming that VLMs treat images as a ‘bag of objects’ and …

Content truncated to 500 words