What Does BERT with Vision Look At?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265–5275 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5265 What Does BERT with Vision Look At? Liunian Harold Li†, Mark Yatskar∗, Da Yin◦, Cho-Jui Hsieh† & Kai-Wei Chang† †University of California, Los Angeles ∗Allen Institute for Artificial Intelligence ◦Peking University liunian.harold.li@cs.ucla.edu, marky@allenai.org, wade yin9712@pku.edu.cn, {chohsieh, kwchang}@cs.ucla.edu Abstract Pre-trained visually grounded language mod- els such as ViLBERT, LXMERT, and UNITER have achieved significant performance im- provement on vision-and-language tasks but what they learn during pre-training remains un- clear. In this work, we demonstrate that cer- tain attention heads of a visually grounded lan- guage model actively ground elements of lan- guage to image regions. Specifically, some heads can map entities to image regions, per- forming the task known as entity grounding. Some heads can even detect the syntactic re- lations between non-entity words and image regions, tracking, for example, associations be- tween verbs and regions corresponding to their arguments. We denote this ability as syntactic grounding. We verify grounding both quanti- tatively and qualitatively, using Flickr30K En- tities as a testbed. 1 Introduction Recently, BERT (Devlin et al., 2019) variants with vision such as ViLBERT (Lu et al., 2019), LXMERT (Tan and Bansal, 2019), and UNITER (Chen et al., 2019) have achieved new records on several vision-and-language reasoning tasks, e.g. VQA (Antol et al., 2015), NLVR2 (Suhr et al., 2019), and VCR (Zellers et al., 2019). These pre-trained visually grounded language models use Transformers (Vaswani et al., 2017) to jointly model words and image regions. They are pre- trained on paired image-text data, where given parts of the input the model is trained to predict the miss- ing pieces. Despite their strong performance, it remains unclear if these models have learned the desired cross-modal representations. Conversely, a large body of work (Liu et al., 2019; Tenney et al., 2019; Clark et al., 2019) has fo- cused on understanding the internal behaviours of pre-trained language models (Peters et al., 2018b; Radford et al., 2018; Devlin et al., 2019) and find that they capture linguistic features such as POS, syntactic structures, and coreferences. This inspires us to ask: what do visually grounded language mod- els learn during pre-training? Following Clark et al. (2019), we find that cer- tain attention heads of a visually grounded lan- guage model acquire an intuitive yet fundamental ability that is often believed to be a prerequisite for advanced visual reasoning (Plummer et al., 2015): grounding of language to image regions. We first observe that some heads can perform entity grounding, where entities that have direct semantic correspondences in the image are mapped to the correct regions. For example, in Figure 1, the word “man” attends to the person on the left of the image. Further, non-entity words often attend to image regions that correspond to their syntactic neighbors and we call this syntactic grounding. For example, “wearing” is attending to its subject, the man in the image. We argue that syntactic grounding …

Content truncated to 500 words