Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9352–9369 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation Xiaohang Tang† Yi Zhou∗ University of Liverpool†, Cardiff University∗, Amazon‡ {sgxtang4, danushka}@liverpool.ac.uk zhouy131@cardiff.ac.uk Danushka Bollegala†,‡ Abstract Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic vari- ations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using time- sensitive templates. Given two snapshots C1 and C2 of a corpus taken respectively at two distinct timestamps T1 and T2, we first pro- pose an unsupervised method to select (a) pivot terms related to both C1 and C2, and (b) an- chor terms that are associated with a specific pivot term in each individual snapshot. We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms. Moreover, we propose an automatic method to learn time-sensitive templates from C1 and C2, without requiring any human super- vision. Next, we use the generated prompts to adapt a pretrained MLM to T2 by fine-tuning using those prompts. Multiple experiments show that our proposed method reduces the per- plexity of test sentences in C2, outperforming the current state-of-the-art. 1 Introduction Contextualised word embeddings produced by MLMs (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Yang et al., 2019) represent the mean- ing of a word with respect to the context in which it appears in and have reported substantial perfor- mance gains in various NLP tasks. The usage of words change over time and the same word might be associated with different words to mean differ- ent concepts over time (Koch, 2016; Baybee, 2015; Rama and Borin, 2015). For example, the word gay has gradually changed its meaning from happy to homosexual over the last five decades (Robinson, 2012; Campbell, 2004). However, MLMs are often trained using a static snapshot of a corpus taken at a specific timestamp, and are not updated afterwards. Because of this reason, existing pretrained MLMs do not capture the temporal semantic variations of words. For example, Loureiro et al. (2022) showed that neither the original version of BERT (Devlin et al., 2019) nor RoBERTa (Liu et al., 2019) are up- to-date with the information related to the current coronavirus pandemic. To address the above-mentioned limitations, we propose a Dynamic Contextualised Word Embed- ding (DCWE) method that adapts a given pre- trained MLM from one timestamp T1 to another T2 using two snapshots of a corpus C1 and C2, sam- pled respectively at times T1 and T2. We represent a word x by an embedding that depends both on the context c of x, as well as on time T. Our word em- beddings are dynamic because they depend on the time, and contextualised because they also depend on the context. We model the problem of adapting a given pre- trained MLM to a specific timestamp T2 as an instance …

Content truncated to 500 words