Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4764 - 4780 July 10-15, 2022 ©2022 Association for Computational Linguistics Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora Xisen Jin†1 Dejiao Zhang2 Henghui Zhu2 Wei Xiao2 Shang-Wen Li‡2 Xiaokai Wei2 Andrew Arnold2 Xiang Ren1 1University of Southern California 2AWS AI Labs {xisenjin,xiangren}@usc.edu {dejiaoz, henghui, weixiaow, shangwenl, xiaokaiw, anarnld} @amazon.com Abstract Pretrained language models (PTLMs) are typi- cally learned over a large, static corpus and fur- ther fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data dis- tributions that deviate from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually up- dated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with dif- ferent continual learning algorithms, and keep track of the downstream task performance (af- ter fine-tuning). We evaluate PTLM’s ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also im- prove knowledge transfer, allowing models to achieve better downstream performance over the latest data, and improve temporal gen- eralization when distribution gaps exist be- tween training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models. 1 Introduction Pretrained language models (PTLMs) have achieved remarkable performance on benchmark datasets for a range of NLP tasks (Liu et al., 2019b; Brown et al., 2020). However, when deployed in the wild, NLP systems must deal with emerging data that have constantly shifting data distribution, different from the text corpora they were initially pretrained on — for example, when new data do- mains are introduced (upper part of Fig. 1) (Gu- rurangan et al., 2020), or when the language uses † Work done during an internship at AWS AI Labs. ‡ Work done while at Amazon. Domain-Incremental Research Paper Stream Knowledge retention All domains Chronologically-Ordered Tweet Stream 2014 2016 2018 2020 Adaptation & Temporal generalization Latest data Bio- Medical Computer Science Materials Science Physics Figure 1: Two data streams created for studying life- long language model pre-training. We focus on evalu- ating knowledge retention on the domain-incremental research papers stream; we focus on adaptation to the latest data and temporal generalization on the chrono- logically ordered tweet stream. and vocabulary change over time (lower part of Fig. 1) (Lazaridou et al., 2021). Fine-tuning from a static and possibly “outdated" PTLM may limit the model performance on downstream tasks, as the PTLM may no longer provide an effective model initialization (Beltagy et al., 2019; Müller et al., 2020). Here we look to understand whether con- tinuously adapting a PTLM to emerging data can yield gains on various downstream …

Content truncated to 500 words