ELLE: Efficient Lifelong Pre-training for Emerging Data

Findings of the Association for Computational Linguistics: ACL 2022, pages 2789 - 2810 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ELLE: Efficient Lifelong Pre-training for Emerging Data Yujia Qin1,2,3∗, Jiajie Zhang1,2,3∗, Yankai Lin4, Zhiyuan Liu1,2,3,5,6†, Peng Li7‡, Maosong Sun1,2,3,5,6,8†, Jie Zhou4 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Beijing National Research Center for Information Science and Technology 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Pattern Recognition Center, WeChat AI, Tencent Inc. 5International Innovation Center of Tsinghua University, Shanghai, China 6Beijing Academy of Artificial Intelligence 7Institute for AI Industry Research (AIR), Tsinghua University, China. 8Jiangsu Collaborative Innovation Center for Language Ability, Xuzhou, China {qyj20,jiajie-z19}@mails.tsinghua.edu.cn Abstract Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of var- ious sources may continuously grow. This re- quires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre- training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specif- ically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM’s width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentan- gle the versatile knowledge learned during pre- training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream perfor- mances. The codes are publicly available at https://github.com/thunlp/ELLE. 1 Introduction Pre-trained language models (PLM) have broken the glass ceiling for various natural language pro- cessing (NLP) tasks (Radford et al., 2018; Devlin et al., 2019; Han et al., 2021). However, most of the existing PLMs are typically trained with a static snapshot of the web information, ignor- ing that in real-world scenarios, streaming data ∗Indicates equal contribution. †Corresponding author. ‡Part of the work was done while Peng Li was working at Tencent. from various sources may continuously grow, e.g., the gatherings of literary works (Zhu et al., 2015), news articles (Zellers et al., 2019) and science pa- pers (Lo et al., 2020). In addition, the distribution of incoming data may also vary over time. This requires PLMs to continually integrate the informa- tion from all the sources to grasp the versatile struc- tural and semantic knowledge comprehensively, so that PLMs could utilize the proper knowledge to boost the performance in various downstream tasks. A simple yet effective way to integrate all the information is to pre-train PLMs on all the existing data exhaustively. However, such a process is com- putationally expensive (Schwartz et al., 2019), es- pecially under the information explosion era when tremendous data is continually collected. This leaves us an important question: with limited com- putational resources, how can we efficiently adapt PLMs in a lifelong manner? We formulate …

Content truncated to 500 words