Knowledge Inheritance for Pre-trained Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3921 - 3937 July 10-15, 2022 ©2022 Association for Computational Linguistics Knowledge Inheritance for Pre-trained Language Models Yujia Qin1,2,3, Yankai Lin4, Jing Yi1,2,3, Jiajie Zhang1,2,3, Xu Han1,2,3, Zhengyan Zhang1,2,3, Yusheng Su1,2,3, Zhiyuan Liu1,2,3,5,6∗, Peng Li7†, Maosong Sun1,2,3,5∗, Jie Zhou4 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Beijing National Research Center for Information Science and Technology 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Pattern Recognition Center, WeChat AI, Tencent Inc. 5International Innovation Center of Tsinghua University, Shanghai, China 6Quan Cheng Laboratory 7Institute for AI Industry Research (AIR), Tsinghua University, China. qyj20@mails.tsinghua.edu.cn Abstract Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of pa- rameters, setting off a wave of training ever- larger PLMs. However, it requires tremen- dous computational resources to train a large- scale PLM, which may be practically unaf- fordable. In addition, existing large-scale PLMs are mainly trained from scratch individ- ually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifi- cally, we introduce a pre-training framework named “knowledge inheritance” (KI) and ex- plore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experi- mental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs’ pre-training settings, includ- ing model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge trans- fer. The implementation is publicly avail- able at https://github.com/thunlp/ Knowledge-Inheritance. 1 Introduction Recently, it has become a consensus in the NLP community to use pre-trained language models (PLMs) as the backbone for various downstream tasks (Han et al., 2021; Min et al., 2021). Despite ∗Corresponding author. †Part of the work was done while Peng Li was working at Tencent. the great follow-up efforts of exploring various pre-training techniques and model architectures, researchers find that simply enlarging the model capacity, data size and training steps can further improve the performance of PLMs (Kaplan et al., 2020; Li et al., 2020b). This discovery sets off a wave of training large-scale PLMs (Raffel et al., 2019; Brown et al., 2020; Fedus et al., 2021). Although huge PLMs have shown awesome per- formance (Bommasani et al., 2021), it requires tremendous computational resources to train large- scale PLMs (Schwartz et al., 2019), raising severe environmental concerns on the prohibitive compu- tational costs. Moreover, existing PLMs are gener- ally trained from scratch individually, ignoring that many well-trained PLMs are available. This leaves us an important question: how could existing PLMs benefit training larger PLMs in future? Considering that humans can leverage the knowl- edge summarized by their predecessors to learn new tasks, so that the learning process could be- come efficient; similarly, it is worth inheriting the implicit knowledge distributed in existing …

Content truncated to 500 words