Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2277–2290 May 2-6, 2023 ©2023 Association for Computational Linguistics Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks Joseph Renner1 Pascal Denis1 Rémi Gilleron1 Angèle Brunellière2 1Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000, Lille, France {firstname.lastname}@inria.fr 2Univ. Lille, CNRS, UMR 9193 - SCALab, F-59000 Lille, France angele.brunelliere@univ-lille.fr Abstract Recent work on predicting category structure with distributional models, using either static word embeddings (Heyman and Heyman, 2019) or contextualized language models (CLMs) (Misra et al., 2021), report low correlations with human ratings, thus calling into question their plausibility as models of human semantic memory. In this work, we revisit this question testing a wider array of methods for probing CLMs for predicting typicality scores. Our ex- periments, using BERT (Devlin et al., 2018), show the importance of using the right type of CLM probes, as our best BERT-based typical- ity prediction methods substantially improve over previous works. Second, our results high- light the importance of polysemy in this task: our best results are obtained when using a dis- ambiguation mechanism. Finally, additional experiments reveal that Information Content- based WordNet (Miller, 1995), also endowed with disambiguation, matches the performance of the best BERT-based method, and in fact cap- tures complementary information, which can be combined with BERT to achieve enhanced typicality predictions. 1 Introduction The empirical success of contextual language mod- els (CLMs) (Peters et al., 2018; Devlin et al., 2018; Liu et al., 2019) has led to much research analyzing their functionality (Rogers et al., 2020; Wu et al., 2020; Ethayarajh, 2019) and how they acquire se- mantic and world knowledge (Petroni et al., 2019). It also raises the question of their plausibility as models of human semantic memory (Chronis and Erk, 2020; Ettinger, 2020; Garí Soler and Apidi- anaki, 2021). The study of categorical knowledge, in particular typicality, provides a window into this question (Murphy, 2004). As initially observed by Rosch (1975), native speakers of English consider that certain exemplars (e.g., robin, crow) are more representative than others (e.g., penguin, ostrich) of a conceptual category (birds). That is, categor- ical knowledge is organized along a graded struc- ture. According to prototype theory (Rosch, 1975), the structure follows from the fact that properties frequently shared among category members tend to be integrated into its prototype. The main question we raise in this paper is whether distributional models, and CLMs in par- ticular, are indeed aware of category structure, as captured in existing human typicality norms for English. The broader question for cognitive sci- ence is whether or not this type of knowledge can be learned through text-based exposure alone, thus contributing to the larger debate on the role of lan- guage in learning semantic knowledge (Lupyan and Lewis, 2019). But there are also more practical mo- tivations for this work, as many NLP tasks (such as information retrieval, question answering, natural language inference) can arguably benefit from typi- cality …

Content truncated to 500 words