Diachronic word embeddings and semantic shifts: a survey

Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397 Santa Fe, New Mexico, USA, August 20-26, 2018. 1384 Diachronic word embeddings and semantic shifts: a survey Andrey Kutuzov Lilja Øvrelid Terrence Szymanski♦ Erik Velldal University of Oslo, Norway {andreku | liljao | erikve}@ifi.uio.no ♦ANZ, Melbourne, Australia terry.szymanski@gmail.com Abstract Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications. 1 Introduction The meanings of words continuously change over time, reflecting complicated processes in language and society. Examples include both changes to the core meaning of words (like the word gay shifting from meaning ‘carefree’ to ‘homosexual’ during the 20th century) and subtle shifts of cultural associations (like Iraq or Syria being associated with the concept of ‘war’ after armed conflicts had started in these countries). Studying these types of changes in meaning enables researchers to learn more about human language and to extract temporal-dependent data from texts. The availability of large corpora and the development of computational semantics have given rise to a number of research initiatives trying to capture diachronic semantic shifts in a data-driven way. Re- cently, word embeddings (Mikolov et al., 2013b) have become a widely used input representation for this task. There are dozens of papers on the topic, mostly published after 2011 (we survey them in Section 3 and further below). However, this emerging field is highly heterogenous. There are at least three different research communities interested in it: natural language processing (and computational linguistics), information retrieval (and computer science in general), and political science. This is re- flected in the terminology, which is far from being standardized. One can find mentions of ‘temporal embeddings,’ ‘diachronic embeddings,’ ‘dynamic embeddings,’ etc., depending on the background of a particular research group. The present survey paper attempts to describe this diversity, introduce some axes of comparison and outline main challenges which the practitioners face. Figure 1 shows the timeline of events that influenced the research in this area: in the following sections we cover them in detail. This survey is restricted in scope to research which traces semantic shifts using distributional word em- bedding models (that is, representing lexical meaning with dense vectors produced from co-occurrence data). We only briefly mention other data-driven approaches also employed to analyze temporal-labeled corpora (for example, topic modeling). Also, we do not cover syntactic shifts and other changes in …

Content truncated to 500 words