bert2BERT: Towards Reusable Pretrained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2134 - 2148 May 22-27, 2022 c⃝2022 Association for Computational Linguistics bert2BERT: Towards Reusable Pretrained Language Models Cheng Chen1†, Yichun Yin2, Lifeng Shang2, Xin Jiang2, Yujia Qin1, Fengyu Wang1, Zhi Wang3,4‡, Xiao Chen2, Zhiyuan Liu1, Qun Liu2 1Department of Computer Science and Technology, Tsinghua University 2Huawei Noah’s Ark Lab, 3Tsinghua Shenzhen International Graduate School 4Peng Cheng Laboratory {c-chen19,qyj20,wangfy20}@mails.tsinghua.edu.cn {yinyichun,shang.lifeng,jiang.xin,chen.xiao2,qun.liu}@huawei.com wangzhi@sz.tsinghua.edu.cn,liuzy@tsinghua.edu.cn Abstract In recent years, researchers tend to pre-train ever-larger language models to explore the up- per limit of deep models. However, large lan- guage model pre-training costs intensive com- putational resources, and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful. In this paper, we propose bert2BERT1, which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and sig- nificantly improve the pre-training efficiency of the large model. Specifically, we extend the pre- vious function-preserving (Chen et al., 2016) method proposed in computer vision on the Transformer-based language model, and fur- ther improve it by proposing a novel method, advanced knowledge for the large model’s ini- tialization. In addition, a two-stage learning method is proposed to further accelerate the pre-training. We conduct extensive experi- ments on representative PLMs (e.g., BERT and GPT) and demonstrate that (1) our method can save a significant amount of training cost com- pared with baselines including learning from scratch, StackBERT (Gong et al., 2019) and MSLT (Yang et al., 2020); (2) our method is generic and applicable to different types of pre- trained models. In particular, bert2BERT saves about 45% and 47% computational cost of pre- training BERTBASE and GPTBASE by reusing the models of almost their half sizes. 1 Introduction Pre-trained language models (PLMs), such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018, 2019; Brown et al., 2020), ELECTRA (Clark et al., 2020), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), have achieved great † This work is done when Cheng Chen is an intern at Huawei Noah’s Ark Lab. ‡ Corresponding author. 1Our code is available at https://github.com/ huawei-noah/Pretrained-Language-Model. 0 1 2 3 4 5 6 7 8 FLOPs (1e19) 1.4 1.6 1.8 2.0 2.2 2.4 2.6 MLM Loss 4 5 6 7 1.40 1.42 1.44 1.46 1.48 1.437 100% 75.7% 54.8% BERTBASE StackBERT bert2BERT Figure 1: Loss curves of bert2BERT and baselines. StackBERT (Gong et al., 2019) is based on the pro- gressive training setting. More details are shown in Table 2. success in natural language processing (NLP). However, the pre-training process of large PLMs can be extremely computationally expensive and produces huge carbon footprints. For example, GPT-3 uses 3.1E+6 GPU hours for training, at an estimated cost of $4.6 million2, consuming a lot of computing resources. Therefore, how to reduce the training cost of PLM is of great importance to Green AI (Schwartz et al., 2020). Recently, there is a trend …

Content truncated to 500 words