Contextualized embeddings for semantic change detection: Lessons learned

Contextualized language models for semantic change detection: lessons learned Andrey Kutuzov, University of Oslo, Norway andreku@ifi.uio.no Erik Velldal, University of Oslo, Norway erikve@ifi.uio.no Lilja Øvrelid, University of Oslo, Norway liljao@ifi.uio.no Abstract We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can ofen predict high change scores for words which are not undergoing any real diachronic semantic shif in the lexicographic sense of the term (or at least the status of these shifs is question- able). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is diferent from the types of issues observed in methods based on static embeddings. Additionally, they ofen merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues. 1 Introduction Lexical semantic change detection (LSCD) is a rela- tively recent sub-field within natural language process- ing. However, comprehensive surveys of data-driven modeling of diachronic semantic change are already available (Tang, 2018; Kutuzov et al., 2018; Tahmasebi et al., 2021a). Dedicated workshops on computational approaches to historical language change took place at the ACL conferences (Tahmasebi et al., 2019, 2021b, 2022) and the results of the SemEval-2020 Task 1 on unsupervised lexical semantic change detection were announced in March 2020 (Schlechtweg et al., 2020). Shared tasks for other languages followed soon (Basile et al., 2020; Kutuzov and Pivovarova, 2021). The majority of the SemEval-2020 shared task par- ticipants employed methods based on word embed- dings of various types. About half of them tried to make use of contextualized (‘token-based’) architec- tures like ELMo (Peters et al., 2018a) or BERT (De- vlin et al., 2019). Although the winning systems still used non-contextualized (‘static’ or ‘type-based’) em- beddings like word2vec (Mikolov et al., 2013), the difer- ence in scores was not dramatic and we are most likely going to see more work in this direction. We agree with Schlechtweg et al. (2020) that as the contextualizing technologies mature, there will be a beter understand- ing of how to properly use them for semantic change related tasks. Indeed, at the RuShifEval shared task on LSCD for Russian (Kutuzov and Pivovarova, 2021), the leader-board was already dominated by contextualized models. The current paper aims to contribute to this im- proved understanding by qualitatively analyzing the output of contextualized embedding-based approaches to the diachronic semantic change detection task. Hence, our work falls into the second category of ground truth semantic change evaluation, as defined by Hengchen et al. (2021): what is evaluated is the ranked output of …

Content truncated to 500 words