When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 123 - 131 July 14-15, 2022 ©2022 Association for Computational Linguistics When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings Elizabeth Soper and Jean-Pierre Koenig Department of Linguistics State University of New York at Buffalo {esoper,jpkoenig}@buffalo.edu Abstract Recent work using word embeddings to model semantic categorization have indicated that static models outperform the more recent con- textual class of models (Majewska et al., 2021). In this paper, we consider polysemy as a possi- ble confounding factor, comparing sense-level embeddings with previously studied static em- beddings on both coarse- and fine-grained cate- gorization tasks. We find that the effect of pol- ysemy depends on how one defines semantic categorization; while sense-level embeddings dramatically outperform static embeddings in predicting coarse-grained categories derived from a word sorting task, they perform ap- proximately equally in predicting fine-grained categories derived from context-free similarity judgments. Our findings highlight the differ- ent processes underlying human behavior on different types of semantic tasks. 1 Introduction A great deal of work has been devoted in recent years to creating computational models of meaning (Landauer and Dumais, 1997; Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018; Devlin et al., 2019). Such models have been evaluated on a variety of semantic tasks, from word pair simi- larity judgments to document classification. One task that has received relatively little attention is semantic categorization. Besides making pair-wise judgments about the similarity between two words, humans can also reason about higher-order struc- tures; we can tell not only that robin and sparrow are similar to each other, for example, but also that they belong in a group with other birds (e.g. ostrich and pigeon). Based on the impressive performance of embedding models on other semantic tasks, we expect such models to excel at identifying semantic categories as well. Our particular interest is on the role of polysemy in semantic categorization. Because words gener- ally have multiple distinct senses, categorization decisions will depend on which sense of a word is being considered. Representing the distinct senses of polysemous words, then, should be important to modeling how humans categorize words. For this reason, we expect contextual embeddings, which represent each instance of a word in context as a unique embedding, to model semantic categoriza- tion better than static models, which conflate every use of a word into a single representation. But, in fact, recent work evaluating different word em- bedding models on verb categorization suggests just the opposite; Majewska et al. (2021) found that contextual models perform poorly compared to older static models. In the following paper, we challenge this result. First, we extend the evaluation from Majewska et al. (2021), who compare word embedding clusters to coarse-grained semantic categories generated by humans in a word sorting task, by evaluating sense- specific embeddings in addition to the static em- beddings previously reported. We find that retain- ing sense-level information from contextual BERT embeddings more than doubles its F1 score, outper- …

Content truncated to 500 words