Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 109–132 November 10–11, 2021. ©2021 Association for Computational Linguistics 109 Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color Mostafa Abdou∗ University of Copenhagen Artur Kulmizev Uppsala University Daniel Hershcovich University of Copenhagen Stella Frank University of Trento Ellie Pavlick Brown University Anders Søgaard University of Copenhagen Abstract Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases — (Paris, Capital, France). However, simple relations of this type can of- ten be recovered heuristically and the extent to which models implicitly reflect topological structure that is grounded in world, such as per- ceptual structure, is unknown. To explore this question, we conduct a thorough case study on color. Namely, we employ a dataset of monolexemic color terms and color chips rep- resented in CIELAB, a color space with a per- ceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text- derived color term representations, we find sig- nificant correspondence. Analyzing the differ- ences in alignment across the color spectrum, we find that warmer colors are, on average, bet- ter aligned to the perceptual color space than cooler ones, suggesting an intriguing connec- tion to findings from recent work on efficient communication in color naming. Further anal- ysis suggests that differences in alignment are, in part, mediated by collocationality and dif- ferences in syntactic usage, posing questions as to the relationship between color perception and usage and context. 1 Introduction Without grounding or interaction with the world, language models (LMs) learn representations that encode various aspects of formal linguistic struc- ture (e.g., morphosyntax (Tenney et al., 2019)) and semantic information (e.g., lexical similarity (Reif et al., 2019a)). Beyond this, it has been sug- gested that text-only training data is enough for LMs to also acquire factual and relational informa- tion about the world (Davison et al., 2019; Petroni et al., 2019). This includes, for instance, some ∗For correspondence: {abdou,soegaard}@di.ku.dk Figure 1: Right: Color orientation in 3d CIELAB space. Left: linear mapping from BERT (CC, see §2) color term embeddings to the CIELAB space. features of concrete and abstract concepts, such as objects’ attributes and affordances (Forbes et al., 2019b; Weir et al., 2020). Furthermore, the rep- resentational geometry of LMs has been found to naturally reflect human lexical similarity and re- latedness judgements, as well as analogy relation- ships (Chronis and Erk, 2020). However, the extent to which these models reflect the structures that exist in humans’ perceptual world—such as the topology of visual perception (Chen, 1982), the structure of the color spectrum (Ennis and Zaidi, 2019; Provenzi, 2020), or of odour spaces (Rossiter, 1996; Chastrette, 1997)—is not well-understood. If LMs are indeed able to capture such topologies—in some domains, at least—it would mean that these structures are a) somehow reflected in language and, thereby, encoded in the textual training data on which models are trained, …

Content truncated to 500 words