exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1433–1439 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1433 exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources Wen Tai H. T. Kung Xin Dong Center for Artificial Intelligence in Medicine, Chang Gung Memorial Hospital iroup9797@cgmh.org.tw Harvard University kung@harvard.edu Harvard University xindong@g.harvard.edu Marcus Comiter Harvard University marcuscomiter@g.harvard.edu Chang-Fu Kuo Center for Artificial Intelligence in Medicine, Chang Gung Memorial Hospital zandis@adm.cgmh.org.tw Abstract We introduce exBERT, a training method to extend BERT pre-trained models from a gen- eral domain to a new pre-trained model for a specific domain with a new additive vocabu- lary under constrained training resources (i.e., constrained computation and data). exBERT uses a small extension module to learn to adapt an augmenting embedding for the new do- main in the context of the original BERT’s embedding of a general vocabulary. The exBERT training method is novel in learning the new vocabulary and the extension mod- ule while keeping the weights of the origi- nal BERT model fixed, resulting in a substan- tial reduction in required training resources. We pre-train exBERT with biomedical ar- ticles from ClinicalKey and PubMed Cen- tral, and study its performance on biomedical downstream benchmark tasks using the MTL- Bioinformatics-2016 dataset. We demonstrate that exBERT consistently outperforms prior approaches when using limited corpus and pre- training computation resources. 1 Introduction Pre-trained language representation models have led to breakthrough performance improvements in downstream natural language processing (NLP) tasks including named entity recognition (Sang and De Meulder, 2003), question answering (Rajpurkar et al., 2016), and sentence classification (Socher et al., 2013). However, pre-trained language mod- els face two challenges as their applications expand: 1) Large Training Resources: Training requires substantial computation and data, see, e.g., BERT- large (Devlin et al., 2018), RoBERTa (Liu et al., 2019). 2) Embedding of Domain-specific Vocab- ulary: A specialized domain, such as the biomedi- cal domain on which this work focuses, has its own vocabulary, and sentences in the domain may have words from both the original language model’s vo- cabulary and new domain-specific vocabulary. Be- ing able to operate on this mixture of vocabulary is essential in achieving high performance on down- stream tasks in the new domain (Garneau et al., 2019). These challenges are particularly pronounced in biomedical domains, where there are many domain- specific words. Prior approaches have addressed these issues by either constructing the pre-trained model from scratch with a new vocabulary (e.g., SciBERT (Beltagy et al., 2019)) or adapting the existing pre-trained model by using it as the initial model in learning vocabulary embeddings for the new domain (e.g., BioBERT (Lee et al., 2019)). However, constructing the model with a new vo- cabulary from scratch requires substantial compu- tational resources and training data. Adapting the existing pre-trained model leads to sub-optimal per- formance on downstream tasks because the original vocabulary may not be proper for biomedical do- mains (Garneau et al., 2019; Hu et al., 2019). We propose exBERT, a novel approach that …

Content truncated to 500 words