Adapting a Language Model While Preserving its General Knowledge

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10177 - 10188 December 7-11, 2022 ©2022 Association for Computational Linguistics Adapting a Language Model While Preserving its General Knowledge Zixuan Ke1, Yijia Shao2, Haowei Lin2, Hu Xu3, Lei Shu1∗ and Bing Liu1 1Department of Computer Science, University of Illinois at Chicago 2Wangxuan Institute of Computer Technology, Peking University 3Meta AI 1{zke4,liub}@uic.edu 2{shaoyj, linhaowei}@pku.edu.cn 3huxu@fb.com Abstract Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end- tasks in the domain can give improved per- formances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This pa- per shows that the existing methods are sub- optimal and proposes a novel method to per- form a more informed adaptation of the knowl- edge in the LM by (1) soft-masking the atten- tion heads based on their importance to best pre- serve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowl- edge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the ef- fectiveness of the proposed approach.1 1 Introduction Pre-trained general-purpose language mod- els (LMs) like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020) have become a standard component in almost all NLP applications. Researchers have also found that domain-adaptive pre-training (or DA-training for short) using an unlabeled corpus in a specific domain to adapt an LM can further improve the end-task performance in the domain (Gururangan et al., 2020; Xu et al., 2019a,b; Sun et al., 2019; Alsentzer et al., 2019). Note that domain-adaptive pre-training is also called post-training (Xu et al., 2019a). Existing DA-training methods simply apply the same pre-training objective, i.e., the mask language ∗Now at Google Research leishu@google.com 1https://github.com/UIC-Liu-Lab/DGA model (MLM) loss, to further train an LM using a domain corpus. These methods are sub-optimal because they do not explicitly identify what should be preserved and what should be updated in the LM by the domain corpus. This paper argues that a good DA-training method has two needs. On the one hand, the gen- eral language knowledge learned in the LM should be preserved as much as possible because the target domain data is typically not large enough to be suf- ficient to learn the general knowledge well. For ex- ample, some words and their contexts may appear infrequently in a particular domain. The knowledge about them cannot be learned accurately based on the domain data alone. When these words and con- texts appear in an end-task, the system will have difficulties. Thus, we need to rely on the knowledge about them in the LM. Since existing DA-training updates the LM …

Content truncated to 500 words