Probing Contextual Language Models for Common Ground with Visual Representations

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5367–5377 June 6–11, 2021. ©2021 Association for Computational Linguistics 5367 Probing Contextual Language Models for Common Ground with Visual Representations Gabriel Ilharco Rowan Zellers Ali Farhadi Hannaneh Hajishirzi Paul G. Allen School of Computer Science & Engineering University of Washington {gamaga,rowanz,ali,hannaneh}@cs.washington.edu Abstract The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only rep- resentations in distinguishing between match- ing and non-matching visual representations. Our findings show that language representa- tions alone provide a strong signal for retriev- ing image patches from the correct object cat- egories. Moreover, they are effective in re- trieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models. 1 Introduction Contextual language models trained on text-only corpora are prevalent in recent natural language processing (NLP) literature (Devlin et al., 2019; Liu et al., 2019b; Lan et al., 2019; Raffel et al., 2019). Understanding what their representations encode has been the goal of a number of recent studies (Belinkov and Glass, 2019; Rogers et al., 2020). Yet, much is left to be understood about whether—or to what extent—these models can en- code visual information. We study this problem in the context of lan- guage grounding (Searle et al., 1984; Harnad, 1990; McClelland et al., 2019; Bisk et al., 2020; Ben- der and Koller, 2020), empirically investigating whether text-only representations can naturally be connected to the visual domain, without explicit visual supervision in pre-training. Figure 1: We introduce a probing mechanism that learns a mapping from contextual language represen- tations to visual features. For a number of contextual language models, we evaluate how useful their repre- sentations are for retrieving matching image patches. We argue that context plays a significant role in this investigation. In language, the ability to form context-dependent representations has shown to be crucial in designing pre-trained language models (Peters et al., 2018; Devlin et al., 2019). This is even more important for studying grounding since many visual properties depend strongly on con- text (Sadeghi and Farhadi, 2011). For instance, a “flying bat” shares very few visual similarities with a “baseball bat”; likewise, a “dog sleeping” looks different from a “dog running”. While align- ments between language representations and visual attributes have attracted past interest (Leong and Mihalcea, 2011; Lazaridou et al., 2014, 2015; Lucy and Gauthier, 2017; Collell Talleda et al., 2017), the role of context has been previously overlooked, leaving many open questions about what visual information contextual language representations encode. In this …

Content truncated to 500 words