Finding Structural Knowledge in Multimodal-BERT

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5658 - 5671 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Finding Structural Knowledge in Multimodal-BERT Victor Milewski1 and Miryam de Lhoneux1,2,3 and Marie-Francine Moens1 1Department of Computer Science, KU Leuven 2Department of Computer Science, University of Copenhagen 3Department of Linguistics and Philology, Uppsala University victor (dot) milewski (at) kuleuven.be Abstract In this work, we investigate the knowledge learned in the embeddings of multimodal- BERT models. More specifically, we probe their capabilities of storing the grammatical structure of linguistic data and the structure learned over objects in visual data. To reach that goal, we first make the inherent struc- ture of language and visuals explicit by a de- pendency parse of the sentences that describe the image and by the dependencies between the object regions in the image, respectively. We call this explicit visual structure the scene tree, that is based on the dependency tree of the language description. Extensive prob- ing experiments show that the multimodal- BERT models do not encode these scene trees. Code available at https://github. com/VSJMilewski/multimodal-probes. 1 Introduction In recent years, contextualized embeddings have become increasingly important. Embeddings cre- ated by the BERT model and its variants have been used to get state-of-the-art performance in many tasks (Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019; Radford and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020). Sev- eral multimodal-BERT models have been devel- oped that learn multimodal contextual embeddings through training jointly on linguistic data and vi- sual data (Lu et al., 2019; Su et al., 2019; Li et al., 2019; Chen et al., 2020). They achieve state-of- the-art results across many tasks and benchmarks, such as Visual Question Answering (Goyal et al., 2017), image and text retrieval (Lin et al., 2014), and Visual Commonsense Reasoning (Suhr et al., 2019).1 BERT and multimodal-BERTs are blackbox models that are not easily interpretable. It is not 1From here on we refer to the text-only BERT models as ’BERT’ and the multimodal-BERT models as ’multimodal- BERTs’. trivial to know what knowledge is encoded in the models and their embeddings. A common method for getting insight into the embeddings of both tex- tual and visual content is probing. Language utterances have an inherent grammati- cal structure that contributes to their meaning. Nat- ural images have a characteristic spatial structure that likewise allows humans to interpret their mean- ing. In this paper we hypothesize that the textual and visual embeddings learned from images that are paired with their descriptions encode structural knowledge of both the language and the visual data. Our goal is to reveal this structural knowledge with the use of probing. More specifically, in order to perform this probing, we first make the inherent structure of language and visuals explicit by a map- ping between a dependency parse of the sentences that describe the image and by the dependency be- tween the object regions in the image, …

Content truncated to 500 words