Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899–3908 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3899 Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View Renfen Hu 1,2,♠ Shen Li 3,♣ Shichen Liang 1,2,♠ ♠ {irishu, shichen}@mail.bnu.edu.cn ♣ shen@deeplycurious.ai 1 Institute of Chinese Information Processing, Beijing Normal University 2 UltraPower-BNU Joint Laboratory for Artificial Intelligence, Beijing Normal University 3 DeeplyCurious.ai Abstract Diachronic word embeddings have been widely used in detecting temporal changes. However, existing methods face the meaning conflation deficiency by representing a word as a single vector at each time period. To ad- dress this issue, this paper proposes a sense representation and tracking framework based on deep contextualized embeddings, aiming at answering not only what and when, but also how the word meaning changes. The exper- iments show that our framework is effective in representing fine-grained word senses, and it brings a significant improvement in word change detection task. Furthermore, we model the word change from an ecological viewpoint, and sketch two interesting sense behaviors in the process of language evolution, i.e. sense competition and sense cooperation. 1 Introduction The meanings of words continuously change over time, reflecting complicated processes in language and society (Kutuzov et al., 2018). With the rapid development of language representation learning, word embeddings have been widely introduced into diachronic linguistic studies. By training and comparing word embeddings of different time epochs, one can capture the semantic drift of words (Kim et al., 2014), learn diachronic analo- gies between terms (Szymanski, 2017), as well as discover the statistical laws of semantic change (Hamilton et al., 2016). Furthermore, this kind of method has gained fruitful results in broader social science studies, e.g. tracing armed con- flicts (Kutuzov et al., 2017), gender and ethnic stereotypes (Garg et al., 2018) and social attitudes (Jaidka et al., 2018). It is well known that word meaning can be rep- resented with a range of senses. However, ex- isting methods only assign one embedding to a 1840 1860 1880 1900 1920 1940 1960 1980 2000 0.0 0.2 0.4 0.6 0.8 1.0 sense_1_adjective: Foolish, stupid, or unimpressive. sense_2_adjective: (of a person) homosexual (used especially of a man) sense_3_noun: A homosexual, especially a man. sense_4_adjective: Light-hearted and carefree. Figure 1: The evolvement of four senses for word gay. Two important phenomena: (1) competition between sense 2 and sense 4; (2) cooperation between sense 2 and sense 3. word for a time period, thus they face challenges in representing senses and tracking the change of them. Given the word embeddings, one can tell the coarse-grained change of the word from one time to another, e.g. the word gay’s nearest neighbors in the vector space move from cheerful and flaunt- ing to homosexual and lesbian. But these word representations are not able to show which sense has changed, which sense is stable, and how they may interact with each other. Recently, an increasing boom on large-scale pre-trained language models e.g. …

Content truncated to 500 words