Task-adaptive Pre-training of Language Models with Word Embedding Regularization

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4546–4553 August 1–6, 2021. ©2021 Association for Computational Linguistics 4546 Task-adaptive Pre-training of Language Models with Word Embedding Regularization Kosuke Nishida, Kyosuke Nishida, Sen Yoshida NTT Media Intelligence Laboratories, NTT Corporation kosuke.nishida.ap@hco.ntt.co.jp Abstract Pre-trained language models (PTLMs) ac- quire domain-independent linguistic knowl- edge through pre-training with massive textual resources. Additional pre-training is effective in adapting PTLMs to domains that are not well covered by the pre-training corpora. Here, we focus on the static word embeddings of PTLMs for domain adaptation to teach PTLMs domain-specific meanings of words. We pro- pose a novel fine-tuning process: task-adaptive pre-training with word embedding regulariza- tion (TAPTER). TAPTER runs additional pre- training by making the static word embeddings of a PTLM close to the word embeddings obtained in the target domain with fastText. TAPTER requires no additional corpus except for the training data of the downstream task. We confirmed that TAPTER improves the per- formance of the standard fine-tuning and the task-adaptive pre-training on BioASQ (ques- tion answering in the biomedical domain) and on SQuAD (the Wikipedia domain) when their pre-training corpora were not dominated by in- domain data. 1 Introduction Pre-trained language models (PTLMs) trained with massive textual and computational resources have achieved high performance in natural language pro- cessing tasks (Devlin et al., 2019). Additional pre- training often is used to tackle domain discrep- ancies between the downstream task and the pre- training corpora. Additional pre-training with a large corpus in the domain of the downstream task, such as BioBERT (Lee et al., 2020), improves the performance on the task (Alsentzer et al., 2019; Beltagy et al., 2019; Chalkidis et al., 2020). How- ever, this approach requires large corpora in the tar- get domain and entails a high computational cost. Gururangan et al. (2020) proposed task-adaptive pre-training (TAPT), which is additional pre- training using only the training data of the down- stream task. TAPT can be regarded as a new fine- tuning process in which the standard fine-tuning is preceded by low-cost additional pre-training. In this study, we focus on the static word em- beddings of PTLMs (i.e., non-contextualized 0-th layer representations) for domain adaptation. Our method is designed to teach PTLMs the domain- specific meanings of the words as static word em- beddings. We are motivated by the observation that the middle BERT layers capture the syntac- tic information (Hewitt and Manning, 2019; Jawa- har et al., 2019; Liu et al., 2019a). We consider that we can adapt the models without harming the domain-independent linguistic knowledge con- tained in higher layers by learning the static word embeddings directly. We propose a novel fine-tuning process called task-adaptive pre-training with word embedding regularization (TAPTER). First, TAPTER obtains word embeddings in the target domain by adapt- ing a pre-trained fastText model (Bojanowski et al., 2017) to the target domain using the training data of the downstream task. Next, TAPTER runs the task- adaptive pre-training by making the static word embeddings of the PTLM close to …

Content truncated to 500 words