Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4811–4819 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4811 Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift Matej Martinc, Petra Kralj Novak, Senja Pollak Jozef Stefan Institute Ljubljana, Slovenia {matej.martinc, petra.kralj.novak, senja.pollak}@ijs.si Abstract We propose a new method that leverages contextual embeddings for the task of diachronic semantic shift detection by generating time specific word representations from BERT embeddings. The results of our experiments in the domain specific LiverpoolFC corpus suggest that the proposed method has performance comparable to the current state-of-the-art without requiring any time consuming domain adaptation on large corpora. The results on the newly created Brexit news corpus suggest that the method can be successfully used for the detection of a short-term yearly semantic shift. And lastly, the model also shows promising results in a multilingual settings, where the task was to detect differences and similarities between diachronic semantic shifts in different languages. Keywords: Contextual embeddings, Diachronic semantic shift, Diachronic news analysis 1. Introduction While language is many times mistakenly perceived as a stable, unchanging structure, it is in fact constantly evolv- ing and adapting to the needs of its users. It is a well re- searched fact that some words and phrases can change their meaning completely in a longer period of time. The word gay, which was a synonym for cheerful until the 2nd half of the 20th century, is just one of the examples found in the lit- erature. On the other hand, we are just recently beginning to research and measure more subtle semantic changes that occur in much shorter time periods. These changes reflect a sudden change in language use due to changes in the politi- cal and cultural sphere or due to the localization of language use in somewhat closed communities. The study of how word meanings change in time has a long tradition (Bloomfield, 1933) but it has only recently saw a surge in popularity and quantity of research due to recent advances in modelling semantic relations with word em- beddings (Mikolov et al., 2013) and increased availability of textual resources. The current state-of-the-art in mod- elling semantic relations are contextual embeddings (De- vlin et al., 2018; Peters et al., 2018), where the idea is to generate a different vector for each context a word appears in, i.e., for each specific word occurrence. This solves the problems with word polysemy and employing this type of embeddings has managed to improve the state-of-the-art on a number of natural language understanding tasks. How- ever, contextual embeddings have not yet been widely em- ployed in the discovery of diachronic semantic shifts. In this study, we present a novel method that relies on contextual embeddings to generate time specific word rep- resentations that can be leveraged for the purpose of di- achronic semantic shift detection 1. We also show that the proposed approach has the following advantages over ex- isting state-of-the-art methods: • It shows …

Content truncated to 500 words