Decoding Symbolism in Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3311–3324 July 9-14, 2023 ©2023 Association for Computational Linguistics Decoding Symbolism in Language Models Meiqi Guo Rebecca Hwa Adriana Kovashka Department of Computer Science, University of Pittsburgh, Pittsburgh PA, USA meiqi.guo@pitt.edu {hwa, kovashka}@cs.pitt.edu Abstract This work explores the feasibility of eliciting knowledge from language models (LMs) to de- code symbolism, recognizing something (e.g., roses) as a stand-in for another (e.g., love). We present our evaluative framework, Symbolism Analysis (SymbA), which compares LMs (e.g., RoBERTa, GPT-J) on different types of symbol- ism and analyzes the outcomes along multiple metrics. Our findings suggest that conventional symbols are more reliably elicited from LMs while situated symbols are more challenging. Results also reveal the negative impact of the bias in pre-trained corpora. We further demon- strate that a simple re-ranking strategy can mit- igate the bias and significantly improve model performances to be on par with human perfor- mances in some cases. 1 Introduction Symbolism is an important literary device that helps to persuade ideas concisely (Symons, 2014). A system that can decode symbolism should rec- ognize that one item (e.g., a baby) is a stand-in for something else (e.g., innocence). It has applica- tions in understanding persuasive texts as well as the visual media (Liu et al., 2022; Guo et al., 2021; Akula et al., 2023). For example, a social media moderator needs to know that certain seemingly be- nign phrase or object may signal some banned be- havior; an intelligent writing tutor should recognize (in)appropriate usages of symbolism in student es- says; a persuasive text/image generator may convey its message more effectively by appropriate uses of symbolism. With these potential applications in mind, this work explores whether state-of-the- art LMs encapsulate enough implicit and abstract knowledge to infer symbolic relationships. Specifi- cally, we ask: given some observed physical object or content (referred to as the signifier), can LMs predict an appropriate corresponding conceptual symbolic reference (referred to as the signified)1? Decoding symbolism is a challenging task (even for humans). First, symbols serve many different purposes, from representing figures of speech and modes of thought to denoting various signs, pass- words, and customs (Jones, 1918). Thus, some types of signifier-signified relationships may be more difficult to decode than others. Prior work suggests that LMs encapsulate some commonsense knowledge (Speer et al., 2017); therefore, we an- ticipate LMs may capture the more semantically related symbolic relationships (e.g., a fork as a sym- bol of food because it is UsedFor eating), but what about those involving a longer chain of reasoning? How do additional factors such as the complexity of the LM and the choice of the prompt impact the performances of different LMs? Second, sym- bols may be situational: the same signifier may be a stand-in for different references under different scenarios. For example, while a baby often repre- sents innocence, when depicted as being held by a harried parent, that baby comes to symbolize bur- den …

Content truncated to 500 words