Downstream Task Performance of BERT Models Pre-Trained Using Automatically De-Identified Clinical Data

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4245–4252 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4245 Downstream Task Performance of BERT Models Pre-Trained Using Automatically De-Identified Clinical Data Thomas Vakili, Anastasios Lamproudis, Aron Henriksson, Hercules Dalianis Department of Computer and Systems Sciences (DSV), Stockholm University, Kista, Sweden {thomas.vakili, anastasios, aronhen, hercules}@dsv.su.se Abstract Automatic de-identification is a cost-effective and straightforward way of removing large amounts of personally identifiable information from large and sensitive corpora. However, these systems also introduce errors into datasets due to their imperfect precision. These corruptions of the data may negatively impact the utility of the de-identified dataset. This paper de-identifies a very large clinical corpus in Swedish either by removing entire sentences containing sensitive data or by replacing sensitive words with realistic surrogates. These two datasets are used to perform domain adaptation of a general Swedish BERT model. The impact of the de-identification techniques is assessed by training and evaluating the models using six clinical downstream tasks. The results are then compared to a similar BERT model domain-adapted using an unaltered version of the clinical corpus. The results show that using an automatically de-identified corpus for domain adaptation does not negatively impact downstream performance. We argue that automatic de-identification is an efficient way of reducing the privacy risks of domain-adapted models and that the models created in this paper should be safe to distribute to other academic researchers. Keywords: Privacy-preserving machine learning, pseudonymization, de-identification, Swedish clinical text, pre-trained language models, BERT, downstream tasks, NER, multi-label classification, domain adaptation 1. Introduction Natural Language Processing (NLP) research is cur- rently dominated by so-called pre-trained language models based on transformers (Vaswani et al., 2017), which were popularized by the introduction of the BERT model by Devlin et al. (2019). These language models typically consist of millions – even billions – of parameters that are learned from enormous corpora. The success of pre-trained language models in general- domain tasks has prompted research into whether these models also succeed in medical-domain tasks. Language models are taught to model language by learning the statistical distributions of the words in their training data. However, words often have different meanings depending on in which domain they are used. The word chest has a dual meaning in everyday lan- guage – something used for storage or a region of the body – but only one of these is relevant in a medical context. A language model which has learned the word chest from a general-domain corpus may have a repre- sentation of the word that is sub-optimal in the medical domain. Indeed, many researchers have found that performance on domain-specific tasks is helped by adapting exist- ing language models or pre-training new models using in-domain data (Lee et al., 2019; Beltagy et al., 2019; Lamproudis et al., 2021; Lamproudis et al., 2022b; Lamproudis et al., 2022a). Better performance means that the models will be more useful in helping medical professionals improve patient outcomes. However, …

Content truncated to 500 words