TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6237 - 6250 December 7-11, 2022 ©2022 Association for Computational Linguistics TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models Joel Jang1,∗ Seonghyeon Ye1,∗ Changho Lee3 Sohee Yang1 Joongbo Shin2 Janghoon Han2 Gyeonghun Kim2 Minjoon Seo1 1KAIST 2LG AI Research 3Korea University {joeljang,seonghyeon.ye,sohee.yang,minjoon}@kaist.ac.kr ckdgh0801@korea.ac.kr {jb.shin,janghoon.han,ghkayne.kim}@lgresearch.ai Abstract Language Models (LMs) become outdated as the world changes; they often fail to per- form tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem be- cause the research community still lacks a co- herent dataset for assessing the adaptability of LMs to frequently-updated knowledge cor- pus such as Wikipedia. To this end, we intro- duce TEMPORALWIKI, a lifelong benchmark for ever-evolving LMs that utilizes the differ- ence between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM’s ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning. The dataset and the code is made available at this link. 1 Introduction Large Language Models (LMs) pretrained on a vast amount of text corpus have shown to be highly ef- fective when finetuned or prompted to perform var- ious downstream tasks (Raffel et al., 2019; Brown et al., 2020; Sanh et al., 2022; Wei et al., 2022). However, most of the datasets used to evaluate these LMs are static benchmarks; the train and test data are both from similar points in time. On the other hand, in the real world, factual knowledge is frequently changed, added, or deprecated. For example, suppose a language model is asked what the most dominant coronavirus variant is (Figure 1). ∗ indicates equal contribution. The answer would have been the Delta variant in the fall of 2021 but has changed to the Omicron variant near the end of 2021. If LMs remain un- changed and are not periodically trained to cope with the changing world, they will be outdated very quickly. This means downstream tasks that directly depend on or are finetuned from the LM will suf- fer from temporal misalignment (Luu et al., 2021; Lazaridou et al., 2021), which refers to the mis- alignment in time between the train and test data. Temporal misalignment becomes a critical prob- lem, especially when using language models for knowledge-intensive tasks such as closed-book question answering (Roberts et al., 2020; Petroni et al., 2021; Jang et al., 2022) since they rely solely on the knowledge stored in their parame- ters. Furthermore, LMs augmented with retrieval mechanism (Guu et al., 2020; Lewis et al., 2020; Borgeaud …

Content truncated to 500 words