Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2881–2898 May 2-6, 2023 ©2023 Association for Computational Linguistics Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views Katerina Margatina♢∗ Shuai Wang† Yogarshi Vyas† Neha Anna John† Yassine Benajiba† Miguel Ballesteros† ♢University of Sheffield †AWS AI Labs k.margatina@sheffield.ac.uk, {wshui,yogarshi,nehajohn,benajiy,ballemig}@amazon.com Abstract Temporal concept drift refers to the problem of data changing over time. In NLP, that would en- tail that language (e.g. new expressions, mean- ing shifts) and factual knowledge (e.g. new con- cepts, updated facts) evolve over time. Focus- ing on the latter, we benchmark 11 pretrained masked language models (MLMs) on a series of tests designed to evaluate the effect of tempo- ral concept drift, as it is crucial that widely used language models remain up-to-date with the ever-evolving factual updates of the real world. Specifically, we provide a holistic framework that (1) dynamically creates temporal test sets of any time granularity (e.g. month, quarter, year) of factual data from Wikidata, (2) con- structs fine-grained splits of tests (e.g. updated, new, unchanged facts) to ensure comprehensive analysis, and (3) evaluates MLMs in three dis- tinct ways (single-token probing, multi-token generation, MLM scoring). In contrast to prior work, our framework aims to unveil how robust an MLM is over time and thus to provide a signal in case it has become outdated, by lever- aging multiple views of evaluation. 1 Introduction In the real world, what people talk about and how they tend to speak and write changes constantly over time. In Natural Language Processing (NLP), this entails a challenging shift of the textual data distribution that is commonly referred to as tempo- ral concept drift. Prior work has identified that pre- trained language models (PLMs) tend to become outdated soon after new topics and concepts are emerging (Lazaridou et al., 2021; Dhingra et al., 2022; Agarwal and Nenkova, 2022; Luu et al., 2022), limiting their capability to be robust to newly generated data. We consider the desiderata of language models’ robustness to temporal drift to be twofold. First, LMs should be well adapted to the dynamic use ∗ Work done during an internship at AWS AI Labs. Figure 1: Querying pretrained MLMs on their knowl- edge about the Prime Minister of the United Kingdom. of language, from the linguistic perspective. Lan- guage changes over time, pronunciations evolve, new words and expressions are borrowed or in- vented, the meaning of old words drifts, and mor- phology develops or decays (Blank, 1999; Traugott and Dasher, 2001; Kulkarni et al., 2015). Second, LMs should be aware of the ever-changing reality of the world, from a factual perspective. Models’ factual knowledge should be up-to-date with new facts and concepts (e.g. Covid-19) to be of use continuously. In this work, we focus on the latter; the temporal robustness of LMs to facts that change over time. In an ideal scenario, we would like to know ex- actly when the factual knowledge of a model …

Content truncated to 500 words