Developing a Clinical Language Model for Swedish: Continued Pretraining of Generic BERT with In-Domain Data

Proceedings of Recent Advances in Natural Language Processing, pages 790–797 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_090 790 Developing a Clinical Language Model for Swedish: Continued Pretraining of Generic BERT with In-Domain Data Anastasios Lamproudis anastasios@dsv.su.se Aron Henriksson aronhen@dsv.su.se Department of Computer and Systems Sciences (DSV) Stockholm University Kista, Sweden Hercules Dalianis hercules@dsv.su.se Abstract The use of pretrained language models, fine- tuned to perform a specific downstream task, has become widespread in NLP. Using a generic language model in specialized do- mains may, however, be sub-optimal due to differences in language use and vocabulary. In this paper, it is investigated whether an exist- ing, generic language model for Swedish can be improved for the clinical domain through continued pretraining with clinical text. The generic and domain-specific language models are fine-tuned and evaluated on three representative clinical NLP tasks: (i) identi- fying protected health information, (ii) assign- ing ICD-10 diagnosis codes to discharge sum- maries, and (iii) sentence-level uncertainty pre- diction. The results show that continued pre- training on in-domain data leads to improved performance on all three downstream tasks, in- dicating that there is a potential added value of domain-specific language models for clinical NLP. 1 Introduction Pretrained language models, trained on a variety of readily accessible and large-scale unlabeled cor- pora, and subsequently fine-tuned on downstream tasks using labeled datasets, have led to substan- tial performance gains across a whole host of NLP tasks. This has contributed to ameliorating a major bottleneck in the development of NLP systems, i.e. the need for access to very large amounts of labeled data for supervised learning. In many cases, obtaining large, task-specific datasets in the form of human-annotated corpora is challenging and prohibitively expensive. As a result, the paradigm of pretraining and fine-tuning has become fundamental for contemporary NLP. In particular, with the introduction of models such as BERT (Devlin et al., 2018), which are based exclusively on self-attention, i.e. Transformers (Vaswani et al., 2017), and leverage transfer learn- ing techniques, language models have become in- creasingly accessible; yet, pretraining language models from scratch requires substantial computa- tional resources. While generic language models, trained and re- leased to the public by resource-rich organizations, can be utilized and fine-tuned to perform a par- ticular downstream NLP task without a need for significant resources – neither computational nor in terms of data – it has been shown that their use in specialized domains may be sub-optimal as a result of differences in, for instance, language use and vocabulary (Lewis et al., 2020; Gururangan et al., 2020). This has motivated efforts to develop domain-specific language models, e.g. SciBERT (Beltagy et al., 2019) and BioBERT (Lee et al., 2020). Specialized language models have been devel- oped either by (i) pretraining a language model with in-domain data from scratch, possibly in com- bination with out-domain data, or by (ii) continu- ing to pretrain an existing, general language model with in-domain data (domain-adaptive pretraining), either by using large amounts of in-domain data, if available, or by only using task-related unlabeled data …

Content truncated to 500 words