Deep Neural Models of Semantic Shift

Proceedings of NAACL-HLT 2018, pages 474–484 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep Neural Models of Semantic Shift Alex Rosenfeld The University of Texas at Austin Department of Linguistics alexbrosenfeld@gmail.com Katrin Erk The University of Texas at Austin Department of Linguistics katrin.erk@mail.utexas.edu Abstract Diachronic distributional models track changes in word use over time. In this pa- per, we propose a deep neural network di- achronic distributional model. Instead of modeling lexical change via a time series as is done in previous work, we represent time as a continuous variable and model a word’s usage as a function of time. Ad- ditionally, we have created a novel syn- thetic task, which quantitatively measures how well a model captures the semantic trajectory of a word over time. Finally, we explore how well the derivatives of our model can be used to measure the speed of lexical change. 1 Introduction Diachronic distributional models have provided interesting insights into how words change mean- ing. Generally, they are used to explore how specific words have changed meaning over time (Sagi et al., 2011; Gulordava and Baroni, 2011; Jatowt and Duh, 2014; Kim et al., 2014; Kulka- rni et al., 2015; Bamler and Mandt, 2017; Hell- rich and Hahn, 2017), but they have also been used to explore historical linguistic theories (Xu and Kemp, 2015; Hamilton et al., 2016a,b), to pre- dict the emergence of novel senses (Bamman and Crane, 2011; Rohrdantz et al., 2011; Cook et al., 2013, 2014), and to predict world events (Kutuzov et al., 2017a,b). Diachronic distributional models are distribu- tional models where the vector for a word changes over time. Thus, we can calculate the cosine sim- ilarity between the vectors for a word at two dif- ferent time points to measure how much that word has changed over time and we can perform a near- est neighbor analysis to understand in what direc- tion a word is changing. For example, diachronic distributional models can detect that the word gay has greatly changed by comparing the word vector for gay across different time points. They can also be used to discover that gay has shifted its mean- ing from happy to homosexual by analyzing when those words show up as nearest neighbors to gay. Previous research in diachronic distributional semantics has used models where data is par- titioned into time bins and a synchronic model is trained on each bin. A synchronic model is a vanilla, time-independent distributional model, such as skip-gram. However, there are several technical issues associated with data binning. For example, if the bins are too large, you can only achieve extremely coarse grained representations of lexical change over time. However, if the bins are too small, the synchronic models get trained on insufficient data. In this paper, we have built the first diachronic distributional model that represents time as a con- tinuous variable instead of employing data bin- ning. There are several advantages to treating time as continuous. The first …

Content truncated to 500 words