Efficiently and Thoroughly Anonymizing a Transformer Language Model for Dutch Electronic Health Records: a Two-Step Method

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1098–1103 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1098 Efficiently and Thoroughly Anonymizing a Transformer Language Model for Dutch Electronic Health Records: a Two-Step Method Stella Verkijk, Piek Vossen VU University Amsterdam De Boelelaan 1105, 1081 HV Amsterdam, Netherlands stellaverkijk@outlook.com, p.t.j.m.vossen@vu.nl Abstract Neural Network (NN) architectures are used more and more to model large amounts of data, such as text data available online. Transformer-based NN architectures have shown to be very useful for language modelling. Although many researchers study how such Language Models (LMs) work, not much attention has been paid to the privacy risks of training LMs on large amounts of data and publishing them online. This paper presents a new method for anonymizing a language model by presenting the way in which MedRoBERTa.nl, a Dutch language model for hospital notes, was anonymized. The two step method involves i) automatic anonymization of the training data and ii) semi-automatic anonymization of the LM’s vocabulary. Adopting the fill-mask task where the model predicts what tokens are most probable to appear in a certain context, it was tested how often the model will predict a name in a context where a name should be. It was shown that it predicts a name-like token 0.2% of the time. Any name-like token that was predicted was never the name originally presented in the training data. By explaining how a LM trained on highly private real-world medical data can be safely published with open access, we hope that more language resources will be published openly and responsibly so the community can profit from them. Keywords: Anonymization, Language Model, Medical Text Data 1. Introduction Deep learning with neural networks (NNs) has taken the machine learning world by storm, becoming the foundation of new AI-based services (Shokri and Shmatikov, 2015; Shickel et al., 2017). BERT by Google (Devlin et al., 2019) and RoBERTa by Face- book (Liu et al., 2019), Transformer-based NNs trained on the enormous amounts of language data available online like Wikipedia pages, have shown enormous success in the modelling of language. Such Language Models (LMs) now serve as the base for many specific NLP systems like Machine Translation, Concept Ex- traction, Question Answering and many more. How- ever, a downside of these language models and NNs in general is that it is not entirely clear yet how they work. They are often referred to as black-box-like systems (Rogers et al., 2020). Among other concerns, this calls for vigilance for privacy breaches. Even if the training data is sourced online and is therefore not seen as pro- tected information, a model trained on this data can be used to systematically search for specific information a malevolent user might be looking for. If the training data was not sourced online and does contain sensitive information, the model itself might be used to recover patterns the model has seen in the training phase. Applying deep learning has also …

Content truncated to 500 words