Dynamic Contextualized Word Embeddings

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6970–6984 August 1–6, 2021. ©2021 Association for Computational Linguistics 6970 Dynamic Contextualized Word Embeddings Valentin Hofmann*‡, Janet B. Pierrehumbert†*, Hinrich Sch¨utze‡ *Faculty of Linguistics, University of Oxford †Department of Engineering Science, University of Oxford ‡Center for Information and Language Processing, LMU Munich valentin.hofmann@ling-phil.ox.ac.uk Abstract Static word embeddings that represent words by a single vector cannot capture the vari- ability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic word em- beddings, we introduce dynamic contextual- ized word embeddings that represent words as a function of both linguistic and extralin- guistic context. Based on a pretrained lan- guage model (PLM), dynamic contextualized word embeddings model time and social space jointly, which makes them attractive for a range of NLP tasks involving semantic vari- ability. We highlight potential application sce- narios by means of qualitative and quantitative analyses on four English datasets. 1 Introduction Over the last decade, word embeddings have rev- olutionized the field of NLP. Traditional methods such as LSA (Deerwester et al., 1990), word2vec (Mikolov et al., 2013a,b), GloVe (Pennington et al., 2014), and fastText (Bojanowski et al., 2017) com- pute static word embeddings, i.e., they represent words as a single vector. From a theoretical stand- point, this way of modeling lexical semantics is problematic since it ignores the variability of word meaning in different linguistic contexts (e.g., poly- semy) as well as different extralinguistic contexts (e.g., temporal and social variation). The first shortcoming was addressed by the in- troduction of contextualized word embeddings that represent words as vectors varying across linguis- tic contexts. This allows them to capture more complex characteristics of word meaning, includ- ing polysemy. Contextualized word embeddings are widely used in NLP, constituting the semantic backbone of pretrained language models (PLMs) such as ELMo (Peters et al., 2018a), BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), XLNet φ(k) ij e(k) ij ˜e(k) d Figure 1: Dynamic contextualized word embeddings. A static embedding ˜e(k) ( ) is mapped to a dynamic embedding e(k) ij ( ) by a function d that takes time and social space into account. The scattered points ( ) are contextualized versions of e(k) ij . Variability in φ(k) ij indi- cates semantic dynamics across time and social space. The embeddings have 768 dimensions. (Yang et al., 2019), ELECTRA (Clark et al., 2020), and T5 (Raffel et al., 2020). A concurrent line of work focused on the second shortcoming of static word embeddings, resulting in various types of dynamic word embeddings. Dy- namic word embeddings represent words as vectors varying across extralinguistic contexts, in particu- lar time (e.g., Rudolph and Blei, 2018) and social space (e.g., Zeng et al., 2018). In this paper, we introduce dynamic contextual- ized word embeddings that combine the strengths of contextualized word embeddings with the flex- ibility of dynamic word embeddings. Dynamic contextualized word …

Content truncated to 500 words