Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1489–1501, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change William L. Hamilton, Jure Leskovec, Dan Jurafsky Department of Computer Science, Stanford University, Stanford CA, 94305 wleif,jure,jurafsky@stanford.edu Abstract Understanding how words change their meanings over time is key to models of language and cultural evolution, but his- torical data on meaning is scarce, mak- ing theories hard to develop and test. Word embeddings show promise as a di- achronic tool, but have not been carefully evaluated. We develop a robust method- ology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evo- lution. Using six historical corpora span- ning four languages and two centuries, we propose two quantitative laws of seman- tic change: (i) the law of conformity—the rate of semantic change scales with an in- verse power-law of word frequency; (ii) the law of innovation—independent of fre- quency, words that are more polysemous have higher rates of semantic change. 1 Introduction Shifts in word meaning exhibit systematic regu- larities (Br´eal, 1897; Ullmann, 1962). The rate of semantic change, for example, is higher in some words than others (Blank, 1999) — com- pare the stable semantic history of cat (from Proto- Germanic kattuz, “cat”) to the varied meanings of English cast: “to mould”, “a collection of actors’, “a hardened bandage”, etc. (all from Old Norse kasta, “to throw”, Simpson et al., 1989). Various hypotheses have been offered about such regularities in semantic change, such as an in- creasing subjectification of meaning, or the gram- maticalization of inferences (e.g., Geeraerts, 1997; Blank, 1999; Traugott and Dasher, 2001). But many core questions about semantic change remain unanswered. One is the role of fre- quency. Frequency plays a key role in other lin- guistic changes, associated sometimes with faster change—sound changes like lenition occur in more frequent words—and sometimes with slower change—high frequency words are more resistant to morphological regularization (Bybee, 2007; Pagel et al., 2007; Lieberman et al., 2007). What is the role of word frequency in meaning change? Another unanswered question is the relationship between semantic change and polysemy. Words gain senses over time as they semantically drift (Br´eal, 1897; Wilkins, 1993; Hopper and Trau- gott, 2003), and polysemous words1 occur in more diverse contexts, affecting lexical access speed (Adelman et al., 2006) and rates of L2 learning (Crossley et al., 2010). But we don’t know whether the diverse contextual use of pol- ysemous words makes them more or less likely to undergo change (Geeraerts, 1997; Winter et al., 2014; Xu et al., 2015). Furthermore, poly- semy is strongly correlated with frequency—high frequency words have more senses (Zipf, 1945; ˙Ilgen and Karaoglan, 2007)—so understanding how polysemy relates to semantic change requires controling for word frequency. Answering these questions requires new meth- ods that can go beyond the case-studies of a few …

Content truncated to 500 words