Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation

Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation Dongha Choi1, HongSeok Choi2, and Hyunju Lee1,2∗ 1Artificial Intelligence Graduate School 2School of Electrical Engineering and Computer Science Gwangju Institute of Science and Technology, Gwangju 61005, South Korea dongha528@gm.gist.ac.kr , {hongking9,hyunjulee}@gist.ac.kr Abstract Since the development and wide use of pre- trained language models (PLMs), several ap- proaches have been applied to boost their per- formance on downstream tasks in specific do- mains, such as biomedical or scientific domains. Additional pre-training with in-domain texts is the most common approach for providing domain-specific knowledge to PLMs. How- ever, these pre-training methods require consid- erable in-domain data and training resources and a longer training time. Moreover, the train- ing must be re-performed whenever a new PLM emerges. In this study, we propose a domain knowledge transferring (DoKTra) framework for PLMs without additional in-domain pre- training. Specifically, we extract the domain knowledge from an existing in-domain pre- trained language model and transfer it to other PLMs by applying knowledge distillation. In particular, we employ activation boundary dis- tillation, which focuses on the activation of hidden neurons. We also apply an entropy reg- ularization term in both teacher training and distillation to encourage the model to generate reliable output probabilities, and thus aid the distillation. By applying the proposed DoKTra framework to downstream tasks in the biomedi- cal, clinical, and financial domains, our student models can retain a high percentage of teacher performance and even outperform the teach- ers in certain tasks. Our code is available at https://github.com/DMCB-GIST/DoKTra. 1 Introduction Recently, transformer (Vaswani et al., 2017)-based language models have been successfully applied in the field of natural language processing (NLP). In particular, the two-stage approach of “pre-training and fine-tuning,” such as BERT (Devlin et al., 2019), has become the standard for NLP applica- tions. Generally, a transformer-based model is pre- trained with a large amount of text data in an unsu- ∗Hyunju Lee is the corresponding author. pervised manner, and then fine-tuned with a small dataset for several downstream tasks. Further, ad- vanced pre-trained language models (PLMs) with improved architectures or training methods con- tinue to emerge, including ALBERT (Lan et al., 2019) or RoBERTa (Liu et al., 2019). However, these models must be further improved for tasks requiring domain knowledge, such as those in the biomedical or financial domains, as the pre-training data usually consist of general do- main text (e.g., Wikipedia). Additional pre-training with in-domain text has been proposed to provide the PLMs with domain-specific knowledge. For ex- ample, in the biomedical domain, several domain- specific PLMs trained with large biomedical texts, such as BioBERT (Lee et al., 2020), PubMedBERT (Gu et al., 2020) and BlueBERT (Peng et al., 2019), have been successfully used as strong baselines for several downstream tasks. Nevertheless, addi- tional pre-training has several limitations, such as the need for sufficient training data and resources, and a longer training time. Furthermore, whenever a new PLM emerges, it must be re-trained to create more advanced domain-specific models. To address …

Content truncated to 500 words