Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1482–1490 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1482 Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA Nina Poerner∗† and Ulli Waltinger† and Hinrich Sch¨utze∗ ∗Center for Information and Language Processing, LMU Munich, Germany †Corporate Technology Machine Intelligence (MIC-DE), Siemens AG Munich, Germany poerner@cis.uni-muenchen.de | inquiries@cislmu.org Abstract Domain adaptation of Pretrained Language Models (PTLMs) is typically achieved by un- supervised pretraining on target-domain text. While successful, this approach is expensive in terms of hardware, runtime and CO2 emis- sions. Here, we propose a cheaper alternative: We train Word2Vec on target-domain text and align the resulting word vectors with the word- piece vectors of a general-domain PTLM. We evaluate on eight English biomedical Named Entity Recognition (NER) tasks and compare against the recently proposed BioBERT model. We cover over 60% of the BioBERT – BERT F1 delta, at 5% of BioBERT’s CO2 footprint and 2% of its cloud compute cost. We also show how to quickly adapt an existing general- domain Question Answering (QA) model to an emerging domain: the Covid-19 pandemic.1 1 Introduction Pretrained Language Models (PTLMs) such as BERT (Devlin et al., 2019) have spearheaded ad- vances on many NLP tasks. Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus. When applying PTLMs to specific domains, it can be useful to domain-adapt them. Domain adap- tation of PTLMs has typically been achieved by pre- training on target-domain text. One such model is BioBERT (Lee et al., 2020), which was initialized from general-domain BERT and then pretrained on biomedical scientific publications. The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recogni- tion (NER) or Question Answering (QA). On the downside, the computational cost of pretraining can be considerable: BioBERTv1.0 was adapted for ten 1www.github.com/npoe/covid-qa days on eight large GPUs (see Table 1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains. We therefore propose a fast, CPU-only domain- adaptation method for PTLMs: We train Word2Vec (Mikolov et al., 2013a) on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM. The PTLM thus gains domain-specific lexi- cal knowledge in the form of additional word vec- tors, but its deeper layers remain unchanged. Since Word2Vec and the vector space alignment are effi- cient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU. In Section 4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1.0) and/or CORD-19 (Covid- 19 Open Research Dataset). We improve over general-domain BERT on eight out of eight biomed- ical NER tasks, using a fraction of the compute cost associated with BioBERT. In Section 5, we show how to quickly adapt an existing Question …

Content truncated to 500 words