Are Language-and-Vision Transformers Sensitive to Discourse? A Case Study of ViLBERT

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 28–38 September 12, 2023 ©2023 Association for Computational Linguistics Are language-and-vision transformers sensitive to discourse? A case study of ViLBERT Ekaterina Voloshina † and Nikolai Ilinykh ∗ † and Simon Dobnik ∗ † ∗Centre for Linguistic Theory and Studies in Probability (CLASP), Department of Philosophy, Linguistics and Theory of Science (FLoV), †University of Gothenburg, Sweden gusvolek@student.gu.se, nikolai.ilinykh@gu.se,simon.dobnik@gu.se Abstract Language-and-vision models have shown good performance on tasks such as image-caption matching and caption generation. However, it is challenging for such models to generate prag- matically correct captions, which adequately reflect what is happening in one or several images. Here we explore to what extent con- textual language-and-vision models are sensi- tive to different discourses, both textual and vi- sual. In particular, we employ one of the multi- modal transformers (ViLBERT) and test if it can match descriptions and images, differentiat- ing them from distractors of different degree of similarity that are sampled from different visual and textual contexts. We place our evaluation in the multi-sentence and multi-image setup, where images and sentences are expected to form a single narrative structure, e.g. discourse. We show that the model can distinguish differ- ent situations but it is not sensitive to differ- ences within one narrative structure. We also demonstrate that the model’s performance de- pends on the task itself, for example, the effect of what modality remains unchanged in non- matching pairs or how similar non-matching pairs are to the original pairs. 1 Introduction Large language models are considered “black boxes” as it is often hard to explain their predic- tions. Therefore, it is essential to evaluate such models on tasks different from downstream ap- plications to see if they have acquired necessary knowledge, including linguistic knowledge, during the pre-training process (Liu et al., 2019; Belinkov, 2022; Elazar et al., 2021). Moreover, model per- formance on linguistic tasks has been shown to correlate with downstream applications (Saphra, 2021). Although the models have been tested on morphology, semantics, and syntax (Conneau et al., 2018; Warstadt et al., 2020; Taktasheva et al., 2021; Sta´nczak et al., 2022; Maudslay and Cot- terell, 2021; Lasri et al., 2022), little has been done on the evaluation of discourse (Hong et al., 2020; Liang et al., 2022). Models such as GPT-3 are good at generating long, coherent sequences but they are often not sensitive to intents or commu- nicative purposes that the narratives are written with (Ruis et al., 2022). In this work, we focus on the evaluation of the sensitivity of multi-modal models to discourse. We understand discourse as contextual information de- fined by the situation of communication connected to social and cultural background. Discourse can be local defined by individual items such as an utterance or an image or narrative defined by a sequence of several items. Discourse meaning in- cludes but is not limited to: • Textual discourse: linguistic relations and de- pendencies that exist between linguistic units across …

Content truncated to 500 words