DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 16207–16221 July 9-14, 2023 ©2023 Association for Computational Linguistics DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains Yanis Labrak∗1,4 Adrien Bazoge∗2,3 Richard Dufour2 Mickael Rouvier1 Emmanuel Morin2 Béatrice Daille2 Pierre-Antoine Gourraud3 1LIA, Avignon Université 2Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France 3Clinique des données, CHU de Nantes, Nantes Université 4Zenidoc {firstname.lastname}@univ-avignon.fr {firstname.lastname}@univ-nantes.fr Abstract In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific do- mains. In this paper, we propose an origi- nal study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomed- ical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained. 1 Introduction During the last years, pre-trained language mod- els (PLMs) have been shown to significantly im- prove performance on many Natural Language Processing (NLP) tasks. Recent models, such as BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019), are more and more taking advantage of the huge quantities of unlabeled data thanks to recent unsupervised approaches like masked lan- guage models based on the Transformers architec- ture (Vaswani et al., 2017). Most of these PLMs are frequently pre-trained on general domain corpora, such as news articles, books or encyclopedia. An additional step of fine-tuning can also be applied to use these PLMs on a targeted task (Devlin et al., 2018). Although these generic models are used in var- ious contexts, recent works have shown that opti- mal performance in specialized domains, such as *Equal contribution. finance (Yang et al., 2020), medical (Yang et al., 2022) or traveling (Zhu et al., 2021), can only be achieved using PLMs adapted to the targeted con- ditions. The adaptation of language models to a domain generally follows two strategies. The first is the training from scratch of a new model using only textual data of the targeted specialty. The second approach, called continual pre-training (Howard and Ruder, 2018), pursues the training of already pre-trained models, allowing them to pass from a generic model to a specialized one. Even if studies have shown that the first strategy generally offers better performance (Lee et al., 2019), the second requires a much more constrained number of re- sources (Chalkidis et al., 2020; El Boukkouri et al., 2022) whether in …

Content truncated to 500 words