G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6585 - 6597 December 7-11, 2022 ©2022 Association for Computational Linguistics G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks Zhongwei Wan1,2∗, Yichun Yin3, Wei Zhang3, Jiaxin Shi4, Lifeng Shang3, Guangyong Chen5†, Xin Jiang3, Qun Liu3 1 Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Science 2University of Chinese Academy of Sciences 3Huawei Noah’s Ark Lab 4Huawei Cloud Computing 5Zhejiang Lab zw.wan1@siat.ac.cn, gychen@zhejianglab.com {yinyichun, zhangwei379, shijiaxin3, shang.lifeng, jiang.xin, qun.liu}@huawei.com Abstract Recently, domain-specific PLMs have been pro- posed to boost the task performance of specific domains (e.g., biomedical and computer sci- ence) by continuing to pre-train general PLMs with domain-specific corpora. However, this Domain-Adaptive Pre-Training (DAPT; Gu- rurangan et al. (2020)) tends to forget the previous general knowledge acquired by gen- eral PLMs, which leads to a catastrophic for- getting phenomenon and sub-optimal perfor- mance. To alleviate this problem, we pro- pose a new framework of General Memory- Augmented Pre-trained Language Model (G- MAP), which augments the domain-specific PLM by a memory representation built from the frozen general PLM without losing any gen- eral knowledge. Specifically, we propose a new memory-augmented layer, and based on it, different augmented strategies are explored to build the memory representation and then adaptively fuse it into the domain-specific PLM. We demonstrate the effectiveness of G-MAP on various domains (biomedical and computer science publications, news, and reviews) and different kinds (text classification, QA, NER) of tasks, and the extensive results show that the proposed G-MAP1 can achieve SOTA results on all tasks. 1 Introduction Pre-trained Language models (PLMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b), have achieved promising perfor- mance on NLP tasks. Typically, these general models are firstly pre-trained on large unlabeled corpus and then directly fine-tuned on downstream tasks. However, there is an inherent gap in text distribution between unlabeled pre-training corpus ∗ This work is done when Zhongwei Wan is an intern at Huawei Noah’s Ark Lab †† Guangyong Chen is the corresponding author 1Code will be available soon. Figure 1: Masked LM (MLM) loss of RoBERTa on 50K randomly sampled documents from each domain before and after DAPT. Figure A and B denote the infer- ence loss of general RoBERTa-base and domain-specific PLMs on the samples of biomedical (BM) and computer science (CS). Figure C means the loss of these models on the samples from the pre-training (PT) corpus of RoBERTa. We report the results of (Gururangan et al., 2020) and lower MLM loss is better. and labeled task corpus, which leads to the dis- tribution shift problem (Gururangan et al., 2020) and makes PLMs perform poorly on some domain tasks (Beltagy et al., 2019; Lee et al., 2020b). To address this shift problem, the domain-adaptive pre- training (DAPT) is proposed (Huang et al., 2019; Beltagy et al., 2019; Gururangan et al., 2020; Lee et al., 2020b) to further pretrain general PLMs …

Content truncated to 500 words