KALA: Knowledge-Augmented Language Model Adaptation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5144 - 5167 July 10-15, 2022 ©2022 Association for Computational Linguistics KALA: Knowledge-Augmented Language Model Adaptation Minki Kang1,2∗ Jinheon Baek1∗ Sung Ju Hwang1,2 KAIST1, AITRICS2 {zzxc1133, jinheon.baek, sjhwang82}@kaist.ac.kr Abstract Pre-trained language models (PLMs) have achieved remarkable success on various nat- ural language understanding tasks. Sim- ple fine-tuning of PLMs, on the other hand, might be suboptimal for domain-specific tasks because they cannot possibly cover knowl- edge from all domains. While adaptive pre-training of PLMs can help them obtain domain-specific knowledge, it requires a large training cost. Moreover, adaptive pre-training can harm the PLM’s performance on the downstream task by causing catastrophic for- getting of its general knowledge. To over- come such limitations of adaptive pre-training for PLM adaption, we propose a novel do- main adaption framework for PLMs coined as Knowledge-Augmented Language model Adaptation (KALA), which modulates the in- termediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts. We validate the per- formance of our KALA on question answer- ing and named entity recognition tasks on mul- tiple datasets across various domains. The results show that, despite being computation- ally efficient, our KALA largely outperforms adaptive pre-training. Code is available at: https://github.com/Nardien/KALA. 1 Introduction Pre-trained Language Models (PLMs) (Devlin et al., 2019; Brown et al., 2020) have shown to be effective on various Natural Language Under- standing (NLU) tasks. Although PLMs aim to ad- dress diverse downstream tasks from various data sources, there have been considerable efforts to adapt the PLMs to specific domains —distributions over the language characterizing a given topic or genre (Gururangan et al., 2020)— for which the ac- quisition of domain knowledge is required to accu- rately solve the downstream tasks (e.g., Biomedical Named Entity Recognition (Dogan et al., 2014)). ∗* Equal contribution 9 10 11 12 Training FLOPs (1016) 66.5 67.0 67.5 68.0 68.5 F1 score KALA (Ours) Task-Adaptive Pre-Training (TAPT) Fine-tuning increasing memory size increasing training steps 182 183 Domain-Adaptive Pre-Training (DAPT) Fine-tuning TAPT DAPT KALA (ours) Figure 1: F1 Score and Training FLOPs for different methods on Question Answering (NewsQA). Note that DAPT uses about 112 times larger data for adaptation. Details are in §5.3 This problem, known as Language Model Adap- tation, can be viewed as a transfer learning prob- lem (Yosinski et al., 2014; Ruder, 2019) under domain shift, where the model is pre-trained on the general domain and the labeled distribution is available for the target domain-specific task. The most prevalent approach to this problem is adaptive pre-training (Figure 2a) which further updates all parameters of the PLM on a large domain-specific or curated task-specific corpus, with the same pre- training strategy (e.g., masked language modeling) before fine-tuning it on the downstream task (Belt- agy et al., 2019; Lee et al., 2020; Gururangan et al., 2020). This continual pre-training of a PLM on the target domain corpus allows it to learn the distri- bution …

Content truncated to 500 words