Are Language Models Sensitive to Semantic Attraction? A Study on Surprisal

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 141–148 July 13-14, 2023 ©2023 Association for Computational Linguistics Are Language Models Sensitive to Semantic Attraction? A Study on Surprisal Yan Cong Purdue University; Feinstein Institutes yancong222@gmail.com Emmanuele Chersoni The Hong Kong Polytechnic University emmanuele.chersoni@polyu.edu.hk Yu-Yin Hsu The Hong Kong Polytechnic University yu-yin.hsu@polyu.edu.hk Alessandro Lenci University of Pisa alessandro.lenci@unipi.it Abstract In psycholinguistics, semantic attraction is a sentence processing phenomenon in which a given argument violates the selectional require- ments of a verb, but this violation is not per- ceived by comprehenders due to its attraction to another noun in the same sentence, which is syntactically unrelated but semantically sound. In our study, we use autoregressive language models to compute the sentence-level and the target phrase-level Surprisal scores of a psy- cholinguistic dataset on semantic attraction. Our results show that the models are sensitive to semantic attraction, leading to reduced Sur- prisal scores, although none of them perfectly matches the human behavioral patterns. 1 Introduction Cases of similarity-based interference have always been at the center of interest for sentence process- ing studies, as they offer strong evidence for cue- based models of memory retrieval during language comprehension (Cunnings and Sturt, 2018). Ac- cording to such accounts, interference emerges be- cause an item with some cues has to be retrieved from memory, and because those cues are simul- taneously matched by multiple items (Van Dyke, 2007; Lewis and Vasishth, 2013). Consider the examples in (1) (Wagers et al., 2009): (1) a. The key to the cells unsurprisingly were rusty. b. The key to the cell unsurprisingly were rusty. Compared to fully grammatical sentences, both elicit longer reading times in humans, but the effect is attenuated in 1a., where there is an attractor (cells) matching the number of the verb, causing an illusion of grammaticality. This phenomenon is known as morphological attraction. Attraction has also been observed at the semantic level, as in the following example from the eye- tracking study by Cunnings and Sturt (2018): (2) a. Julia saw the beer that the lady with the meal quite happily ate during an expensive night out. b. Julia saw the beer that the lady with the wine quite happily ate during an expensive night out. Again, both sentences are implausible, because beer violates the selectional restrictions of the verb ate, but the authors of the study observed that (2a) was processed faster than (2b), due to the presence of a semantically fitting noun (meal) that gener- ates a semantic illusion. Both types of illusion are facilitatory interferences, as they attenuate the effects of anomalies leading to higher costs for the human language processing system. This is a case of semantic attraction. The recent literature in Natural Language Pro- cessing (NLP), on the other hand, has shown an increasing interest in using the Surprisal scores (Hale, 2001; Levy, 2008) computed with Neu- ral Language Models (NLMs) to account for sentence processing phenomena (Futrell et al., 2018; Van Schijndel and …

Content truncated to 500 words