Scalable and Interpretable Semantic Change Detection

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4642–4652 June 6–11, 2021. ©2021 Association for Computational Linguistics 4642 Scalable and Interpretable Semantic Change Detection Syrielle Montariol∗ LISN - CNRS, Univ. Paris-Saclay Societé Générale syrielle.montariol@limsi.fr Matej Martinc∗ Jozef Stefan Institute matej.martinc@ijs.si Lidia Pivovarova University of Helsinki lidia.pivovarova@helsinki.fi Abstract Several cluster-based methods for semantic change detection with contextual embeddings emerged recently. They allow a fine-grained analysis of word use change by aggregating embeddings into clusters that reflect the differ- ent usages of the word. However, these meth- ods are unscalable in terms of memory con- sumption and computation time. Therefore, they require a limited set of target words to be picked in advance. This drastically limits the usability of these methods in open exploratory tasks, where each word from the vocabulary can be considered as a potential target. We pro- pose a novel scalable method for word usage- change detection that offers large gains in pro- cessing time and significant memory savings while offering the same interpretability and better performance than unscalable methods. We demonstrate the applicability of the pro- posed method by analysing a large corpus of news articles about COVID-19. 1 Introduction Studying language evolution is important for many applications, since it can reflect changes in the po- litical and social sphere. In the literature, the study of language evolution either focuses on long-term changes in the meaning of a word, or on more common short-term evolutionary phenomena, such as the word suddenly appearing in a new context, while keeping its meaning unchanged in a lexico- graphic sense. We refer to all types of language evolution—short- or long-term, with or without meaning change—as word usage change, a broad category that includes semantic change, but also any shifts in the context in which a word appears. Recent studies (Giulianelli et al., 2020; Martinc et al., 2020a) show that clustering of contextual em- beddings could be a proxy for word usage change: if clusters, which in theory capture distinct word us- ages, are distributed differently across time periods, ∗ These authors contributed equally. it indicates a possible change in word’s context or even loss or gain of a word sense. Thus, the cluster-based approach offers a more intuitive in- terpretation of word usage change than alternative methods, which look at the neighborhood of a word in each time period to interpret the change (Gonen et al., 2020; Martinc et al., 2020b) and ignore the fact that a word can have more than one meaning. The main limitation of the cluster-based methods is the scalability in terms of memory consumption and time: clustering is applied to each word in the corpus separately and all occurrences of a word need to be aggregated into clusters. For large cor- pora with large vocabularies, where some words can appear millions of times, the use of these meth- ods is severely limited. To avoid the scalability issue, cluster-based meth- ods are generally applied to a …

Content truncated to 500 words