Empirical Study of Diachronic Word Embeddings for Scarce Data

Proceedings of Recent Advances in Natural Language Processing, pages 795–803, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_092 795 Empirical Study of Diachronic Word Embeddings for Scarce Data Syrielle Montariol LIMSI - CNRS Univ. Paris-Sud, Univ. Paris-Saclay Soci´et´e G´en´erale syrielle.montariol@limsi.fr Alexandre Allauzen LIMSI - CNRS Univ. Paris-Sud, Univ. Paris-Saclay alexandre.allauzen@limsi.fr Abstract Word meaning change can be inferred from drifts of time-varying word embeddings. However, temporal data may be too sparse to build robust word embeddings and to dis- criminate significant drifts from noise. In this paper, we compare three models to learn di- achronic word embeddings on scarce data: in- cremental updating of a Skip-Gram from Kim et al. (2014), dynamic filtering from Bamler and Mandt (2017), and dynamic Bernoulli em- beddings from Rudolph and Blei (2018). In particular, we study the performance of differ- ent initialisation schemes and emphasise what characteristics of each model are more suitable to data scarcity, relying on the distribution of detected drifts. Finally, we regularise the loss of these models to better adapt to scarce data. 1 Introduction In all languages, word usage can evolve over time, mirroring cultural or technological evolution of society (Aitchison, 2001). For example, the word ”Katrina” used to be ex- clusively a first name until year 2005 when hurri- cane Katrina devastated the United States coasts. After that tragedy, this word started to be associ- ated with the vocabulary of natural disasters. In linguistics, diachrony refers to the study of temporal variations in the use and meaning of a word. Detecting and understanding these changes can be useful for linguistic research, but also for many tasks of Natural Language Processing (NLP). Nowadays, a growing number of historical textual data is digitised and made publicly avail- able. It can be analysed in parallel with contempo- rary documents, for tasks ranging from text classi- fication to information retrieval. However, the use of conventional word embeddings methods have the drawback to average in one vector the different word’s usages observed across the whole corpus. This static representation hypothesis turns out to be limited in the case of temporal datasets. Assuming that a change in the context of a word mirrors a change in its meaning or usage, a solution is to explore diachronic word embed- dings: word vectors varying through time, follow- ing the changes in the global context of the word. While many authors proposed diachronic embed- ding models these last years, these methods usu- ally need large amounts of data to ensure robust- ness. However, temporal datasets often face the prob- lem of scarcity; beyond the usual scarcity problem of domain-specific corpora or low-resource lan- guages, a temporal dataset can have too short com- pared to the volume of the full dataset.1 Moreover the amount of digital historical texts is limited for many languages, particularly for oldest time peri- ods. This paper addresses the following question: In case of scarce data, how to efficiently learn time-varying word embeddings? For this pur- pose, we compare three diachronic methods on several sizes of datasets. …

Content truncated to 500 words