Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8485–8497, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8485 Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection Adam Tsakalidis1,3 Maria Liakata1,2,3 1Queen Mary University of London, London, UK 2University of Warwick, Coventry, UK 3The Alan Turing Institute, London, UK {atsakalidis,mliakata}@turing.ac.uk Abstract Semantic change detection concerns the task of identifying words whose meaning has changed over time. Current state-of-the-art ap- proaches operating on neural embeddings de- tect the level of semantic change in a word by comparing its vector representation in two distinct time periods, without considering its evolution through time. In this work, we pro- pose three variants of sequential models for de- tecting semantically shifted words, effectively accounting for the changes in the word repre- sentations over time. Through extensive ex- perimentation under various settings with syn- thetic and real data we showcase the impor- tance of sequential modelling of word vectors through time for semantic change detection. Finally, we compare different approaches in a quantitative manner, demonstrating that tem- poral modelling of word representations yields a clear-cut advantage in performance. 1 Introduction Identifying words whose lexical meaning has changed over time is a primary area of research at the intersection of natural language processing and historical linguistics. Through the evolution of language, the task of “semantic change detection” (Tahmasebi et al., 2018; Tang, 2018; Kutuzov et al., 2018) can provide valuable insights on cultural evo- lution over time (Michel et al., 2011). Measuring linguistic change is also relevant to understand- ing the dynamics in online communities (Danescu- Niculescu-Mizil et al., 2013) and the evolution of individuals (McAuley and Leskovec, 2013). Re- cent years have seen a surge in interest in this area since researchers are now able to leverage the in- creasing availability of historical corpora in digital form and develop models that detect the shift in a word’s meaning through time. However, two key challenges in the field still remain. Firstly, there is little work in existing lit- erature on model comparison (Schlechtweg et al., 2019; Dubossarsky et al., 2019; Shoemark et al., 2019). Partially due to the lack of (longitudinal) labelled datasets, existing work assesses model per- formance mainly in a qualitative manner, without quantitative comparisons against prior work. There- fore, it becomes difficult to assess what constitutes an appropriate approach for semantic change de- tection. Secondly, on a methodological front, a large body of related work detects semantically shifted words by pairwise comparisons of their rep- resentations in distinct time periods, ignoring the sequential modelling aspect of the task. Since se- mantic change is a time-sensitive process (Tsaka- lidis et al., 2019), considering consecutive vector representations through time – instead of two bins of word representations (Schlechtweg et al., 2018, 2020) – can be crucial to improving model perfor- mance (Shoemark et al., 2019). Here we tackle both challenges by approaching semantic change detection as an anomaly identifica- tion task. Working on embedding representations of words in …

Content truncated to 500 words