Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15974–15990 July 9-14, 2023 ©2023 Association for Computational Linguistics Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning Shivaen Ramshetty∗ and Gaurav Verma∗ and Srijan Kumar Georgia Institute of Technology {sramshetty3, gverma, srijan}@gatech.edu Abstract The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but do so without leveraging the cross-modal information present in multimodal data. Information from the visual modality, such as color, size, and shape, provide additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., “girl on a chair” → “little girl on a wooden chair”). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of ∼ 15% in MRR and ∼ 20% in F1 score, respectively. Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples. We release the code to encourage robustness eval- uations of deep vision-and-language models: https://github.com/claws-lab/ multimodal-robustness-xmai. 1 Introduction The ability to model the interaction of information in vision and language modalities powers several web applications — text-to-image search (He et al., 2016), summarizing multimodal content (Zhu et al., 2018), visual question answering (Antol et al., *Equal contribution. A street with a mechanical sign with directions. Original Caption Corresponding Image A street with a mechanical sign with directions. small neon Augmented Caption using XMAI (Our) Image is retrieved at rank 104 Image is retrieved at rank 506 ↓ A street with a mechanical sign with directions. busy Augmented Caption using CLARE (Text-only) Image is retrieved at rank 104 Figure 1: We propose Cross-Modal Attribute Insertions (XMAI) — an approach that leverages cross-modal in- teractions in multimodal data to obtain meaningful text augmentations that methods using text-only information (e.g., CLARE) cannot provide. These augmentations highlight vulnerabilities of multimodal models; in this case, the corresponding image is retrieved at a worse rank (104 → 506) for the modified caption. 2015), and editing images using language com- mands (Shi et al., 2021). Ensuring satisfactory user experience within such applications necessi- tates the development of multimodal models that can robustly process text and image data, jointly. Existing research has demonstrated the brittle reasoning mechanism of text-only and image-only models by introducing variations in the inputs (Ev- timov et al., 2020; Li et al., 2021a). Furthermore, prior work have established controlled generation methods for text (Ross et al., 2022), including counterfactuals for model assessment (Madaan et al., …

Content truncated to 500 words