Time Waits for No One! Analysis and Challenges of Temporal Misalignment

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944 - 5958 July 10-15, 2022 ©2022 Association for Computational Linguistics Time Waits for No One! Analysis and Challenges of Temporal Misalignment Kelvin Luu1 Daniel Khashabi2 Suchin Gururangan1 Karishma Mandyam1 Noah A. Smith1,2 1University of Washington 2Allen Institute for AI {kellu,sg01,krm28,nasmith}@cs.washington.edu, danielk@allenai.org Abstract When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting tempo- ral misalignment can degrade end-task perfor- mance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and re- views) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain- specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal mis- alignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously re- ported. We also find that, while temporal adap- tation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.1 1 Introduction Changes in the ways a language is used over time are widely attested (Labov, 2011; Altmann et al., 2009; Eisenstein et al., 2014); how these changes will affect NLP systems built from text corpora, and in particular their long-term performance, is not as well understood. This paper focuses on temporal misalignment, i.e., where training and evaluation datasets are drawn from different periods of time. In today’s pretraining-finetuning paradigm, this misalignment can affect a pretrained language model—a situa- tion that has received recent attention (Jaidka et al., 2018; Lazaridou et al., 2021; Peters et al., 2018; Raffel et al., 2020; Röttger and Pierrehumbert, 2021)—or the finetuned task model, or both. We 1Data and code are available here. suspect that the effects of temporal misalignment will vary depending on the genre or domain of the task’s text, the nature of that task or application, and the specific time periods. We focus primarily on measuring the extent of temporal misalignment on task performance. We consider eight tasks, each with datasets that span at least five years (§2.4), ranging from summarization to entity typing, a subproblem of entity recognition (Borthwick, 1999). Notably, these task datasets span four different domains: social media, scien- tific articles, news, and reviews. We introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time. Our research questions are: (Q1) how does temporal misalignment affect downstream tasks over time? (Q2) how does sensitivity to temporal misalign- ment vary with text domain and task? (Q3) how does temporal misalignment affect lan- guage models across domains and spans of time? (Q4) how effective …

Content truncated to 500 words