A Psycholinguistic Analysis of BERT’s Representations of Compounds

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2230–2241 May 2-6, 2023 ©2023 Association for Computational Linguistics A Psycholinguistic Analysis of BERT’s Representations of Compounds Lars Buijtelaar University of Amsterdam lars.buijtelaar@student.uva.nl Sandro Pezzelle ILLC, University of Amsterdam s.pezzelle@uva.nl Abstract This work studies the semantic representations learned by BERT for compounds, that is, ex- pressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human se- mantic intuitions when dealing with expres- sions (e.g., sunlight) whose overall meaning depends—to a various extent—on the seman- tics of the constituent words (sun, light). We leverage a dataset that includes human judg- ments on two psycholinguistic measures of compound semantic analysis: lexeme mean- ing dominance (LMD; quantifying the weight of each constituent toward the compound mean- ing) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents’ semantics). We show that BERT-based measures moder- ately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for ‘standard’ words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide in- sights into how speakers represent compounds. 1 Introduction Compounds such as sunlight or bodyguard are an interesting benchmark to probe the semantic rep- resentations learned by any NLP models. On the one hand, compounds that are part of a language lexicon (i.e., lexicalized compounds; Gagné and Spalding, 2006) have their own (sets of) established meaning(s). As such, they are lexical items just like any other word. On the other hand, the semantic status of compounds is special since their meaning is the result of the combination of the meaning of two words (hence, the constituents). According to psycholinguistic evidence, this semantic rela- tion does not disappear with lexicalization. Indeed, speakers actively combine constituent meanings when processing both novel and lexicalized com- pounds (Gagné and Spalding, 2009; Ji et al., 2011; Marelli and Luzzatti, 2012; Marelli et al., 2014). In this work, we argue that NLP systems capable of faithfully representing word meanings should account for these aspects. For example, to acknowl- edge that the meaning of handgun relies more on the semantics of gun than of hand (indeed, a hand- gun is a type of gun). Or, that the meaning of sun- light is more directly recoverable from the seman- tics of its constituents (it is more transparent) than is the meaning of muskrat (which is very opaque). Transformer-based encoders such as BERT (De- vlin et al., 2019) are shown to produce word rep- resentations that align well with human semantic intuitions, particularly at their lower layers (Bom- masani et al., 2020; Vuli´c et al., 2020). This sug- gests that these models are effective in encoding the meaning of …

Content truncated to 500 words