An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4854–4864, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4854 An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training Kristjan Arumae, Qing Sun, & Parminder Bhatia Amazon Seattle, USA {arumae, qinsun, parmib}@amazon.com Abstract Pre-training large language models has be- come a standard in the natural language pro- cessing community. Such models are pre- trained on generic data (e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the same domain. However, in order to achieve state-of-the-art performance on out of domain tasks such as clinical named entity recognition and relation extraction, additional in domain pre-training is required. In practice, staged multi-domain pre-training presents per- formance deterioration in the form of catas- trophic forgetting (CF) when evaluated on a generic benchmark such as GLUE. In this paper we conduct an empirical investigation into known methods to mitigate CF. We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. Furthermore, we explore gradient and latent clustering based data selection techniques to improve coverage when using elastic weight consolidation and experience replay methods. 1 Introduction Transformer (Vaswani et al., 2017) based lan- guage modeling has taken over many previous pre- training and initialization approaches (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2019; Liu et al., 2019). Fine-tuning using these architectures yields state-of-the-art results in the order of a few hours. The caveat to these models is that the initial training can be on the scale of many days if not weeks, distributed across multiple GPUs (Strubell et al., 2019), a costly endeavour. Pre-trained language models are adapted to per- form strongly in more specific domains as well. For example, while the original BERT models (Devlin et al., 2019) were trained on English Wikipedia articles and BooksCorpus (Zhu et al., 2015), the Figure 1: Traditional approaches (top) train independent do- main specific language models (newswire, bio-medical, and clinical) which share no cross domain knowledge. They are further fine-tuned on their respective in-domain tasks. Our approach (bottom) shows how several domains are introduced in sequence, with knowledge retention using mitigation tech- niques across all domains. Here the final model has the capa- bility to properly fine-tune on any domain specific task. same masked language modeling was continued on bio-medical data. BioBERT (Lee et al., 2019) was trained using Pubmed abstracts and full arti- cles, meanwhile Clinical BERT (Alsentzer et al., 2019) was further refined using MIMIC-III clinical notes (Johnson et al., 2016). Evidence suggest that understanding the syntactic structure of scientific literature and clinical data from pre-training boosts performance in their respective downstream tasks (Peng et al., 2019). Pre-training is performed with the expectation of building robust, high capacity generalized language models which continue to absorb new domain knowledge. Unfortunately, continual learning (Ring, 1997) suffers from catastrophic forgetting (McCloskey and Cohen, 1989; Ratcliff, 1990) when incorporat- ing domain data in …

Content truncated to 500 words