Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2626–2633 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2626 Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain Hicham El Boukkouri1, Olivier Ferret2, Thomas Lavergne1, Pierre Zweigenbaum1 1Universit´e Paris Saclay, CNRS, LISN, France 2Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France {elboukkouri,lavergne,pz}@lisn.fr, olivier.ferret@cea.fr Abstract BERT models used in specialized domains all seem to be the result of a simple strategy: initializing with the original BERT and then resuming pre-training on a specialized corpus. This method yields rather good performance (e.g. BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), BlueBERT (Peng et al., 2019)). However, it seems reasonable to think that training directly on a specialized corpus, using a specialized vocabulary, could result in more tailored embeddings and thus help performance. To test this hypothesis, we train BERT models from scratch using many configurations involving general and medical corpora. Based on evaluations using four different tasks, we find that the initial corpus only has a weak influence on the performance of BERT models when these are further pre-trained on a medical corpus. Keywords: word embeddings, contextualized embeddings, BERT, medical, biomedical, specialized domain, domain adaptation. 1. Introduction Recent years have witnessed the widespread use of trans- fer learning techniques in Natural Language Processing (NLP) (Pan and Yang, 2010; Ruder, 2019). In fact, after being successful in speech processing (Wang and Zheng, 2015) and computer vision (He et al., 2017), transfer learning was applied in NLP as well with methods like ULMFiT (Howard and Ruder, 2018) showing that pre- trained language models can be successfully adapted through a sequential transfer learning process to bring performance gains to text classification. Shortly after, Radford et al. (2018) generalized these results to other tasks like natural language inference, text comprehen- sion, etc. Following the excitement around transfer learning in NLP, word embedding models moved from the static embedding paradigm (e.g. GloVe (Pennington et al., 2014), fastText (Bojanowski et al., 2017)) to the more dynamic contextual embeddings (ELMo (Peters et al., 2018), BERT (Devlin et al., 2019)) that are able to com- pute context-dependent word representations. Despite the performance gains that are brought by such models (see GLUE benchmark), using them in practice is sig- nificantly more costly than earlier static methods1. As a result, NLP practitioners tend to rely on pre-trained versions rather than training these models from scratch. Today, the most popular contextual embedding model seems to be BERT and the BERT-like models, for which there are pre-trained versions both for the general do- main and for more specialized domains (e.g. BioBERT (Lee et al., 2020), BlueBERT (Peng et al., 2019), Clini- 1Pre-training BERT from scratch may require multiple GPUs for multiple weeks. The benchmarks performed by NVIDIA and available here give a good idea about the cost of the training and the fine-tuning of BERT models. calBERT (Alsentzer et al., 2019a), and PubMedBERT (Gu et al., 2021) for the …

Content truncated to 500 words