Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1538 - 1555 December 7-11, 2022 ©2022 Association for Computational Linguistics Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? Mitja Nikolaus1,2 mitja.nikolaus@univ-amu.fr Emmanuelle Salin1 and Stephane Ayache1 and Abdellah Fourtassi1 and Benoit Favre1 1Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, France 2Aix-Marseille Univ, CNRS, LPL, Aix-en-Provence, France Abstract Recent advances in vision-and-language model- ing have seen the development of Transformer architectures that achieve remarkable perfor- mance on multimodal reasoning tasks. Yet, the exact capabilities of these black-box models are still poorly understood. While much of previ- ous work has focused on studying their ability to learn meaning at the word-level, their ability to track syntactic dependencies between words has received less attention. We take a first step in closing this gap by cre- ating a new multimodal task targeted at evalu- ating understanding of predicate-noun depen- dencies in a controlled setup. We evaluate a range of state-of-the-art models and find that their performance on the task varies consider- ably, with some models performing relatively well and others at chance level. In an effort to explain this variability, our analyses indicate that the quality (and not only sheer quantity) of pretraining data is essential. Additionally, the best performing models leverage fine-grained multimodal pretraining objectives in addition to the standard image-text matching objectives. This study highlights that targeted and con- trolled evaluations are a crucial step for a pre- cise and rigorous test of the multimodal knowl- edge of vision-and-language models. 1 Introduction Vision-and-language (V&L) models have recently shown substantial improvement on a range of mul- timodal reasoning tasks. Taking inspiration from successes in text-only Natural Language Process- ing (Devlin et al., 2019; Brown et al., 2020), state- of-the-art V&L models are usually composed of a Transformer-based architecture pre-trained in a self-supervised manner on large-scale data, and then fine-tuned on downstream tasks. While these models show remarkable perfor- mance on a range of tasks, more controlled and systematic analyses are necessary in order to obtain Target sentence: A man is wearing a hat. Distractor Sentence: A man is wearing glasses. Figure 1: We evaluate V&L models on their ability to track predicate-noun dependencies that require a joint understanding of the linguistic and visual modalities. The task is to find the correct sentence (choosing be- tween the target and distractor) that corresponds to the scene in the image. In this example, the models should connect the predicate “is wearing a hat” to “man”. A model that does not track dependencies would judge the distractor sentence “A man is wearing glasses” as equally likely, as there is a man is the image, as well as a person that is wearing glasses. a better understanding of their exact multimodal knowledge. A range of studies has investigated their abil- ity to map words to their visual referents for nouns (Kazemzadeh et al., 2014; Mao et al., 2016; Shekhar et al., 2017) and verbs (Ronchi and Per- ona, 2015; Yatskar et al., 2016; …

Content truncated to 500 words