Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 148–157 November 10–11, 2021. ©2021 Association for Computational Linguistics 148 Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms Casey Kennington Department of Computer Science Boise State University caseykennington@boisestate.edu Abstract Language models are trained only on text de- spite the fact that humans learn their first lan- guage in a highly interactive and multimodal environment where the first set of learned words are largely concrete, denoting physical entities and embodied states. To enrich lan- guage models with some of this missing ex- perience, we leverage two sources of informa- tion: (1) the Lancaster Sensorimotor norms, which provide ratings (means and standard de- viations) for over 40,000 English words along several dimensions of embodiment, and which capture the extent to which something is expe- rienced across 11 different sensory modalities, and (2) vectors from coefficients of binary clas- sifiers trained on images for the BERT vocab- ulary. We pre-trained the ELECTRA model and fine-tuned the RoBERTa model with these two sources of information then evaluate us- ing the established GLUE benchmark and the Visual Dialog benchmark. We find that en- riching language models with the Lancaster norms and image vectors improves results in both tasks, with some implications for robust language models that capture holistic linguis- tic meaning in a language learning context. 1 Introduction Children learn their first spoken language in a highly interactive setting where generally the first words children learn are concrete words that de- note physical objects, which is an important devel- opmental step in child first language acquisition (Kuperman et al., 2012a; McCune, 2008; Clark, 2013). This is partly because handling the Symbol Grounding Problem–the ablity to connect symbolic knowledge of language with representations of the physical world (Harnad, 1990)–must take place be- fore children learn more abstract concepts later in their cognitive development (Borghi et al., 2019; Ponari et al., 2018). Importantly, the physical world is not just the visual world; children learn that words ground into proprioperceptive states (e.g., a hand grasp around an object has specific muscle activations tied to the word grab), interoceptive states (i.e., affect and valence), as well as all other sensory modalities (e.g., the word stinky grounds into olfactory, the word loud grounds into audi- tory). These claims are evidenced in a large body of child development and cognitive science literature. Smith and Gasser (2005), for example, identified that babies’ experience of the world is profoundly multimodal: babies live in a physical world full of rich regularities that organize perception, action and thought; babies learn in a social world to learn a shared linguistic communicative system that is symbolic. Furthermore, a growing body of litera- ture from linguistics and computational linguistics makes a strong case that the process of language learning (indeed, general human cognition) is em- bodied, interactive, and enacted; i.e., movement in the world is required (Pulverm¨uller, 1999; Lakoff and Johnson, 1999; Barsalou, 2008; Johnson, 2008; Smith and Samuelson, 2009; Di Paolo …

Content truncated to 500 words